youtube-dl: Failed to parse JSON

Question

I posted a question on this yesterday to the GitHub support page and it got flagged as a duplicate - the original answer is here. This issue doesn't automatically fix like it did for that user. Instead, it seems to come and go with no pattern, so I don't have a good way to replicate it. Some songs will work at one point in time, then they won't a couple minutes later.

Error:

[debug] Encodings: locale cp1252, fs utf-8, out UTF-8, pref cp1252
[debug] youtube-dl version 2020.09.20
[debug] Python version 3.7.8 (CPython) - Windows-10-10.0.19041-SP0
[youtube:search] query "iron man 3 song": Downloading page 1
[debug] exe versions: none
[debug] Proxy map: {}
ERROR: query "song name": Failed to parse JSON  caused by JSONDecodeError('Expecting value: line 1 column 1 (char 0)')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

I get the issue when attempting to extract the data from the video. Here is a snippet of the code I am using:

ydlOps = {
    'format': 'bestaudio/best',
    'outtmpl': './%(title)s.webm',
    'noplaylist': True,
    'extractaudio': True,
    'audioformat': 'webm',
    'default_search': 'ytsearch1',
    'quite': True,
    'verbose':True,
    'version': True
}

        with youtube_dl.YoutubeDL(ydlOps) as downloader:
            songData = downloader.extract_info(url, download=download)

I have changed the options, tried other options said to have worked, and nothing seems to make a difference. Some will work, then not, then they will again.

Answer 1

I think this is a youtube-dl bug. I wrote a parser for youtube searches and it also broke. When looking previously at the response from youtube, all the JSON data was stored like this:

window["ytInitialData"] = {...}

So you just had to search through the server's response for the string 'window["ytInitialData"]' to find the relevant JSON and extract it. But now, youtube stores the JSON like this inside the html file sent by the server

var ytInitialData = {...}

This needs to be changed on youtube-dl's part when parsing the results.

What's strange is that sometimes youtube uses the previous version and sometimes it uses the current one. I think it's because the change in javascript is progressively being rolled out accross all youtube servers.

Also note that now, the line containing all the JSON ends with '; ' instead of just ';'. This might also require a change from youtube-dl.

You need to submit a pull request to youtube-dl or wait for somebody to fix it.