YouTube Data API v3 returns rarely suddenly commentsDisabled error which involves an unwanted method switch #20

New Issue

Benjamin_Loison · 2023-01-07T15:52:47+01:00

Benjamin_Loison commented

2023-01-07 15:52:47 +01:00

1: Found error in JSON at URL: https://yt.lemnoslife.com/noKey/commentThreads?part=snippet,replies&videoId=yzuwQlcGde8&maxResults=100&pageToken=QURTSl9pMTJnR2NSRTZLMVY1M0pvTjRlVENLRDhOVWQxbWtmWXRXSERvRVF0RklyZU9sd29ydUhNRWJxVEJXWUxlR0xTTWlaTnR5UlpNYjY= for content:

{
  "error": {
    "code": 400,
    "message": "The API server failed to successfully process the request. While this can be a transient error, it usually indicates that the request's input is invalid. Check the structure of the \u003ccode\u003ecommentThread\u003c/code\u003e resource in the request body to ensure that it is valid.",
    "errors": [
      {
        "message": "The API server failed to successfully process the request. While this can be a transient error, it usually indicates that the request's input is invalid. Check the structure of the \u003ccode\u003ecommentThread\u003c/code\u003e resource in the request body to ensure that it is valid.",
        "domain": "youtube.commentThread",
        "reason": "processingFailure",
        "location": "body",
        "locationType": "other"
      }
    ]
  }
}

`1: Found error in JSON at URL:` https://yt.lemnoslife.com/noKey/commentThreads?part=snippet,replies&videoId=yzuwQlcGde8&maxResults=100&pageToken=QURTSl9pMTJnR2NSRTZLMVY1M0pvTjRlVENLRDhOVWQxbWtmWXRXSERvRVF0RklyZU9sd29ydUhNRWJxVEJXWUxlR0xTTWlaTnR5UlpNYjY= `for content:` ```json { "error": { "code": 400, "message": "The API server failed to successfully process the request. While this can be a transient error, it usually indicates that the request's input is invalid. Check the structure of the \u003ccode\u003ecommentThread\u003c/code\u003e resource in the request body to ensure that it is valid.", "errors": [ { "message": "The API server failed to successfully process the request. While this can be a transient error, it usually indicates that the request's input is invalid. Check the structure of the \u003ccode\u003ecommentThread\u003c/code\u003e resource in the request body to ensure that it is valid.", "domain": "youtube.commentThread", "reason": "processingFailure", "location": "body", "locationType": "other" } ] } } ```

Benjamin_Loison commented

2023-01-07 16:15:38 +01:00

It doesn't seem that the transient error is the problem in fact, as I found out (using below Python script) that a commentsDisabled error happened.

https://yt.lemnoslife.com/noKey/commentThreads?part=snippet,replies&allThreadsRelatedToChannelId=UCWeg2Pkate69NFdBeuRFTAw&maxResults=100&pageToken=QURTSl9pMmFFZnRjTWdXMUhiWlpxTEprNGxBZXMtb3lsQVNMM2Qxa0thVmpaU2Z1cU15ZDRuZXhyY05QcVRmX0MyUDI1ZmxtM3FjU211SjNfWFpoV2FVMkhrX1lVN3BvQ3ZXN25HSXBiN0MtaGp3QjJvWEhja2xGVzRybVQxc1FvWWwyRGdaZ2tsaVZ1a0c5RTM2ZjJtVVNFZy1MeEl5anI0QlFSakVwTGcwYmw5N2Yza0xhTkJ2enNtbXl1ZjVDaFpjZmhZVU5xTXdjdzNLSG1qX2VLUQ==

{
  "error": {
    "code": 403,
    "message": "The video identified by the \u003ccode\u003e\u003ca href=\"/youtube/v3/docs/commentThreads/list#videoId\"\u003evideoId\u003c/a\u003e\u003c/code\u003e parameter has disabled comments.",
    "errors": [
      {
        "message": "The video identified by the \u003ccode\u003e\u003ca href=\"/youtube/v3/docs/commentThreads/list#videoId\"\u003evideoId\u003c/a\u003e\u003c/code\u003e parameter has disabled comments.",
        "domain": "youtube.commentThread",
        "reason": "commentsDisabled",
        "location": "videoId",
        "locationType": "parameter"
      }
    ]
  }
}

So this error during the process might have switch from commentThreads?allThreadsRelatedToChannelid=UC... to playlistItems?playlistId=UU... and commentThreads?videoId=....
Re-executing the HTTP request solves the error...
So we are going to assume that the first HTTP request never leads to a false positive.

findWhenARollbackHappened.py:

#!/usr/bin/python3

import os, json, time, datetime

channelDirectory = 'channels/UCWeg2Pkate69NFdBeuRFTAw/'

requestsLen = len(os.listdir(channelDirectory))
#print(requestsLen)

def getTimestampFromDateString(dateString):
    return int(time.mktime(datetime.datetime.strptime(dateString, "%Y-%m-%dT%H:%M:%SZ").timetuple()))

lastItemTime = int(time.time())

for requestsIndex in range(requestsLen):
    if requestsIndex % 1000 == 0:
        print(f"{requestsIndex} / {requestsLen}")
    with open(f"{channelDirectory}{requestsIndex}.json") as f:
        content = "\n".join(f.read().split("\n")[1:])
        data = json.loads(content)
        if not 'kind' in data:
            print(requestsIndex, data)
        if data['kind'] == 'youtube#commentThreadListResponse':
            firstItemTimeStr = data['items'][0]['snippet']['topLevelComment']['snippet']['publishedAt']
            firstItemTime = getTimestampFromDateString(firstItemTimeStr)
            #print(firstItemTime)
            if lastItemTime < firstItemTime:
                print("A time rollback happened, as we went from {lastItemTime} to {firstItemTime}!")
                exit(1)

It doesn't seem that the transient error is the problem in fact, as I found out (using below Python script) that a `commentsDisabled` error happened. https://yt.lemnoslife.com/noKey/commentThreads?part=snippet,replies&allThreadsRelatedToChannelId=UCWeg2Pkate69NFdBeuRFTAw&maxResults=100&pageToken=QURTSl9pMmFFZnRjTWdXMUhiWlpxTEprNGxBZXMtb3lsQVNMM2Qxa0thVmpaU2Z1cU15ZDRuZXhyY05QcVRmX0MyUDI1ZmxtM3FjU211SjNfWFpoV2FVMkhrX1lVN3BvQ3ZXN25HSXBiN0MtaGp3QjJvWEhja2xGVzRybVQxc1FvWWwyRGdaZ2tsaVZ1a0c5RTM2ZjJtVVNFZy1MeEl5anI0QlFSakVwTGcwYmw5N2Yza0xhTkJ2enNtbXl1ZjVDaFpjZmhZVU5xTXdjdzNLSG1qX2VLUQ== ```json { "error": { "code": 403, "message": "The video identified by the \u003ccode\u003e\u003ca href=\"/youtube/v3/docs/commentThreads/list#videoId\"\u003evideoId\u003c/a\u003e\u003c/code\u003e parameter has disabled comments.", "errors": [ { "message": "The video identified by the \u003ccode\u003e\u003ca href=\"/youtube/v3/docs/commentThreads/list#videoId\"\u003evideoId\u003c/a\u003e\u003c/code\u003e parameter has disabled comments.", "domain": "youtube.commentThread", "reason": "commentsDisabled", "location": "videoId", "locationType": "parameter" } ] } } ``` So this error during the process might have switch from `commentThreads?allThreadsRelatedToChannelid=UC...` to `playlistItems?playlistId=UU...` and `commentThreads?videoId=...`. Re-executing the HTTP request solves the error... So we are going to assume that the first HTTP request never leads to a false positive. `findWhenARollbackHappened.py`: ```py #!/usr/bin/python3 import os, json, time, datetime channelDirectory = 'channels/UCWeg2Pkate69NFdBeuRFTAw/' requestsLen = len(os.listdir(channelDirectory)) #print(requestsLen) def getTimestampFromDateString(dateString): return int(time.mktime(datetime.datetime.strptime(dateString, "%Y-%m-%dT%H:%M:%SZ").timetuple())) lastItemTime = int(time.time()) for requestsIndex in range(requestsLen): if requestsIndex % 1000 == 0: print(f"{requestsIndex} / {requestsLen}") with open(f"{channelDirectory}{requestsIndex}.json") as f: content = "\n".join(f.read().split("\n")[1:]) data = json.loads(content) if not 'kind' in data: print(requestsIndex, data) if data['kind'] == 'youtube#commentThreadListResponse': firstItemTimeStr = data['items'][0]['snippet']['topLevelComment']['snippet']['publishedAt'] firstItemTime = getTimestampFromDateString(firstItemTimeStr) #print(firstItemTime) if lastItemTime < firstItemTime: print("A time rollback happened, as we went from {lastItemTime} to {firstItemTime}!") exit(1) ```

Benjamin_Loison changed title from ~~On YouTube Data API v3 transient error, it seems that there is a rollback~~ to YouTube Data API v3 returns rarely suddenly `commentsDisabled` error which involves an unwanted method switch

2023-01-07 16:20:00 +01:00

Benjamin_Loison added the

quick

bug

high priority

labels 2023-01-07 16:23:07 +01:00

Benjamin_Loison referenced this issue from a commit

2023-01-08 15:43:12 +01:00

Fix #20: YouTube Data API v3 returns rarely suddenly `commentsDisabled` error which involves an unwanted method switch

Benjamin_Loison closed this issue

2023-01-08 15:43:12 +01:00

Benjamin_Loison referenced this issue from a commit