diff --git a/The-crawler.md b/The-crawler.md index 070c3f7..79d97dc 100644 --- a/The-crawler.md +++ b/The-crawler.md @@ -58,7 +58,7 @@ The architecture of the description is the same as the `High overview` one. - [`CHANNELS`](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L311) by using [YouTube operational API Channels: list endpoint with `part=channels`](https://stackoverflow.com/a/74213174) - [`COMMUNITY`](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L342) by using [YouTube operational API Channels: list endpoint with `part=community`](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L346) then using [YouTube operational API Community: list](https://github.com/Benjamin-Loison/YouTube-operational-API/blob/7ff59b2d477c8d2caf6813a114f4201791627cc1/community.php) endpoint to retrieve comments on community posts and proceed with [YouTube operational API CommentThreads: list](https://github.com/Benjamin-Loison/YouTube-operational-API/blob/7ff59b2d477c8d2caf6813a114f4201791627cc1/commentThreads.php) endpoint to proceed to comments pagination - [`PLAYLISTS`](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L402), we proceed with [YouTube operational API Channels: list endpoint with `part=playlists`](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L406) then we proceed with [YouTube Data API v3 PlaylistItems: list](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L418) endpoint and we focus on [videos that are unlisted](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L429), as we can't find them in anyway, that way we treat the video (so notably its comments) and as other playlists, we look for new channels in [the authors of the playlist videos](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L438) - - [`LIVES`](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L468), here we focus on livestream chat messages on not comments. We first retrieve channel videos thanks to [YouTube Data API v3 PlaylistItems: list](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L474) endpoint, then we proceed with [YouTube Data API v3 Videos: list endpoint](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L484) [to check which videos are livestreams](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L488), then there are two possibilities: + - [`LIVE`](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L468), here we focus on livestream chat messages on not comments. We first retrieve channel videos thanks to [YouTube Data API v3 PlaylistItems: list](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L474) endpoint, then we proceed with [YouTube Data API v3 Videos: list endpoint](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L484) [to check which videos are livestreams](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L488), then there are two possibilities: - [the livestreams are ongoing](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L493) and [we proceed](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L496) with [YouTube Livestreaming API LiveChatMessages: list](https://developers.google.com/youtube/v3/live/docs/liveChatMessages/list) endpoint - the livestreams are ended and [we proceed](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/4e162e34c3a5debad1ca6bcbf02701b2c4faa431/main.cpp#L512) with [YouTube operational API LiveChats: list](https://github.com/Benjamin-Loison/YouTube-operational-API/blob/7ff59b2d477c8d2caf6813a114f4201791627cc1/liveChats.php) endpoint