Also modified compression command, as I got `sh: 1: zip: Argument list too long` when compressing the 248,868 json files of the French most subscribers channel.
As got:
```
terminate called after throwing an instance of 'nlohmann::detail::parse_error'
terminate called recursively
what(): [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal
terminate called recursively
```
Tested with `UCWIdqSQekeGmUWlSFeCiEnA` which treated correctly the 36 comments of the only comments enabled video `3F8dFt8LsXY`.
Note that this commit doesn't support comments disabled channels with more than 20,000 videos.
More precisely I used following algorithm with these three channels:
channel id | 1st method | 2nd method
-------------------------|-----------------------|-----------
UCt5USYpzzMCYhkirVQGHwKQ | 16 | 16
UCUo1RqYV8tGjV38sQ8S5p9A | 58,165 | 58,165
UCWIdqSQekeGmUWlSFeCiEnA | *error* (as expected) | 27
```py
"""
Algorithm comparing comments count using:
1. CommentThreads: list with allThreadsRelatedToChannelId filter
2. PlaylistItems: list and CommentThreads: list
Note that the second approach isn't *atomic*, so counts will differ if some comments are posted while retrieving data.
"""
import requests, json
CHANNEL_ID = 'UC...'
API_KEY = 'AIzaSy...'
def getJSON(url, firstTry = True):
if firstTry:
url = 'https://www.googleapis.com/youtube/v3/' + url + f'&key={API_KEY}'
try:
content = requests.get(url).text
except:
print('retry')
return getJSON(url, False)
data = json.loads(content)
return data
items = []
pageToken = ''
while True:
# After having verified, I confirm that using `allThreadsRelatedToChannelId` doesn't return comments of the `COMMUNITY` tab
data = getJSON(f'commentThreads?part=id,snippet,replies&allThreadsRelatedToChannelId={CHANNEL_ID}&maxResults=100&pageToken={pageToken}')
items += data['items']
# In fact once we have top level comment, then with both methods if the replies *count* is correct, then we are fine as we both use the same Comments: list endpoint
"""for item in data['items']:
if 'replies' in item:
if len(item['replies']['comments']) >= 5:
print('should consider replies too!')"""
print(len(items))
if 'nextPageToken' in data:
pageToken = data['nextPageToken']
else:
break
print(len(items))
PLAYLIST_ID = 'UU' + CHANNEL_ID[2:]
videoIds = []
pageToken = ''
while True:
data = getJSON(f'playlistItems?part=snippet&playlistId={PLAYLIST_ID}&maxResults=50&pageToken={pageToken}')
for item in data['items']:
videoIds += [item['snippet']['resourceId']['videoId']]
print(len(videoIds))
if 'nextPageToken' in data:
pageToken = data['nextPageToken']
else:
break
print(len(videoIds))
items = []
for videoIndex, videoId in enumerate(videoIds):
pageToken = ''
while True:
data = getJSON(f'commentThreads?part=id,snippet,replies&videoId={videoId}&maxResults=100&pageToken={pageToken}')
if 'items' in data:
items += data['items']
# repeat replies check as could be the case here and not there
"""for item in data['items']:
if 'replies' in item:
if len(item['replies']['comments']) >= 5:
print('should consider replies too!')"""
print(videoIndex, len(videoIds), len(items))
if 'nextPageToken' in data:
pageToken = data['nextPageToken']
else:
break
print(len(items))
```
As we want to retrieve as many comments as possible, we have to proceed video per video, as [`3F8dFt8LsXY`](https://www.youtube.com/watch?v=3F8dFt8LsXY) for instance has comments but using YouTube Data API v3 CommentThreads: list endpoint with `allThreadsRelatedToChannelId` filter returns for `UCWIdqSQekeGmUWlSFeCiEnA`:
```json
{
"error": {
"code": 403,
"message": "The video identified by the \u003ccode\u003e\u003ca href=\"/youtube/v3/docs/commentThreads/list#videoId\"\u003evideoId\u003c/a\u003e\u003c/code\u003e parameter has disabled comments.",
"errors": [
{
"message": "The video identified by the \u003ccode\u003e\u003ca href=\"/youtube/v3/docs/commentThreads/list#videoId\"\u003evideoId\u003c/a\u003e\u003c/code\u003e parameter has disabled comments.",
"domain": "youtube.commentThread",
"reason": "commentsDisabled",
"location": "videoId",
"locationType": "parameter"
}
]
}
}
```
