> ## Content Index > Fetch the complete content index at: https://serpapi.com/blog/llms.txt > Use this file to discover other available public pages before exploring further. # Scrape YouTube video page with Python - URL: https://serpapi.com/blog/scrape-youtube-video-page-with-python/ - Published: 2022-08-18T15:22:14.000Z - Updated: 2024-02-19T23:37:44.000Z - Description: This blog post is about scraping data from a YouTube video page using Python. - Author: Artur Chukhrai - Tags: YouTube, Web Scraping, Python In this article, we'll learn how to scrape a Youtube video page using a DIY solution. > Update: Since this post published, we've added a new [Youtube API](https://serpapi.com/youtube-search-api) and a [Youtube video API](https://serpapi.com/youtube-video-api). Here is a new blog post on [how to scrape YouTube search results with a simple API using Python](https://serpapi.com/blog/how-to-scrape-youtube-data-with-simple-api/) . ## Intro ~~Currently, we don't have an API that supports extracting data from YouTube video page.~~ This blog post is to show you way how you can do it yourself with provided DIY solution below while we're working on releasing our proper API. The solution can be used for personal use as it doesn't include the [Legal US Shield](https://serpapi.com/#features) that we offer for our paid [production and above plans](https://serpapi.com/pricing) and has its limitations such as the need to bypass blocks, for example, CAPTCHA. You can check our public roadmap to track the progress for this API: ๐Ÿ—บ๏ธ [\[New API\] Support YouTube Video Page Results API](https://github.com/serpapi/public-roadmap/issues/233) ## What will be scraped ![video-page](https://user-images.githubusercontent.com/81998012/183314169-590fd954-0683-4e82-bd18-b90202a4ea98.png) ๐Ÿ“ŒNote: For now, we don't have an API that supports extracting data from individual YouTube page. This blog post is to show you way how you can do it yourself while we're working on releasing our proper API in a meantime. We'll update you on our Twitter once this API will be released. ## Full Code If you don't need explanation, have a look at [full code example in the online IDE](https://replit.com/@chukhraiartur/blog-youtube-video-page#main.py). ```python import re, json, time from selenium import webdriver from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager from parsel import Selector def scroll_page(url): service = Service(ChromeDriverManager().install()) options = webdriver.ChromeOptions() options.add_argument("--headless") options.add_argument('--lang=en') options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36") driver = webdriver.Chrome(service=service, options=options) driver.get(url) old_height = driver.execute_script(""" function getHeight() { return document.querySelector('ytd-app').scrollHeight; } return getHeight(); """) while True: driver.execute_script("window.scrollTo(0, document.querySelector('ytd-app').scrollHeight)") time.sleep(2) new_height = driver.execute_script(""" function getHeight() { return document.querySelector('ytd-app').scrollHeight; } return getHeight(); """) if new_height == old_height: break old_height = new_height selector = Selector(driver.page_source) driver.quit() return selector def scrape_all_data(selector): youtube_video_page = [] all_script_tags = selector.css('script').getall() title = selector.css(".title .ytd-video-primary-info-renderer::text").get() # https://regex101.com/r/gHeLwZ/1 views = int(re.search(r"(.*)\s", selector.css(".view-count::text").get()).group().replace(",", "")) # https://regex101.com/r/9OGwJp/1 likes = int(re.search(r"(.*)\s", selector.css("#top-level-buttons-computed > ytd-toggle-button-renderer:first-child #text::attr(aria-label)").get()).group().replace(",", "")) date = selector.css("#info-strings yt-formatted-string::text").get() duration = selector.css(".ytp-time-duration::text").get() # https://regex101.com/r/0JNma3/1 keywords = "".join(re.findall(r'"keywords":\[(.*)\],"channelId":".*"', str(all_script_tags))).replace('\"', '').split(",") # https://regex101.com/r/9VhH1s/1 thumbnail = re.findall(r'\[{"url":"(\S+)","width":\d*,"height":\d*},', str(all_script_tags))[0].split('",')[0] channel = { # https://regex101.com/r/xFUzq5/1 "id": "".join(re.findall(r'"channelId":"(.*)","isOwnerViewing"', str(all_script_tags))), "name": selector.css("#channel-name a::text").get(), "link": f'https://www.youtube.com{selector.css("#channel-name a::attr(href)").get()}', "subscribers": selector.css("#owner-sub-count::text").get(), "thumbnail": selector.css("#img::attr(src)").get(), } description = selector.css(".ytd-expandable-video-description-body-renderer span:nth-child(1)::text").get() hash_tags = [ { "name": hash_tag.css("::text").get(), "link": f'https://www.youtube.com{hash_tag.css("::attr(href)").get()}' } for hash_tag in selector.css(".ytd-expandable-video-description-body-renderer a") ] # https://regex101.com/r/onRk9j/1 category = "".join(re.findall(r'"category":"(.*)","publishDate"', str(all_script_tags))) comments_amount = int(selector.css("#count .count-text span:nth-child(1)::text").get().replace(",", "")) comments = [] for comment in selector.css("#contents > ytd-comment-thread-renderer"): comments.append({ "author": comment.css("#author-text span::text").get().strip(), "link": f'https://www.youtube.com{comment.css("#author-text::attr(href)").get()}', "date": comment.css(".published-time-text a::text").get(), "likes": comment.css("#vote-count-middle::text").get().strip(), "comment": comment.css("#content-text::text").get(), "avatar": comment.css("#author-thumbnail #img::attr(src)").get(), }) suggested_videos = [] for video in selector.css("ytd-compact-video-renderer"): suggested_videos.append({ "title": video.css("#video-title::text").get().strip(), "link": f'https://www.youtube.com{video.css("#thumbnail::attr(href)").get()}', "channel_name": video.css("#channel-name #text::text").get(), "date": video.css("#metadata-line span:nth-child(2)::text").get(), "views": video.css("#metadata-line span:nth-child(1)::text").get(), "duration": video.css("#overlays #text::text").get().strip(), "thumbnail": video.css("#thumbnail img::attr(src)").get(), }) youtube_video_page.append({ "title": title, "views": views, "likes": likes, "date": date, "duration": duration, "channel": channel, "keywords": keywords, "thumbnail": thumbnail, "description": description, "hash_tags": hash_tags, "category": category, "suggested_videos": suggested_videos, "comments_amount": comments_amount, "comments": comments, }) print(json.dumps(youtube_video_page, indent=2, ensure_ascii=False)) def main(): url = "https://www.youtube.com/watch?v=fbh3OAw3VVQ" result = scroll_page(url) scrape_all_data(result) if __name__ == "__main__": main() ``` ## Preparation **Install libraries**: ```lang-none pip install parsel selenium webdriver webdriver_manager ``` **Reduce the chance of being blocked** Make sure you're using [request headers](https://docs.python-requests.org/en/master/user/quickstart/#custom-headers) [user-agent](https://developer.mozilla.org/en-US/docs/Glossary/User%5Fagent) to act as a "real" user visit. Because default `requests` `user-agent` is [python-requests](https://github.com/psf/requests/blob/589c4547338b592b1fb77c65663d8aa6fbb7e38b/requests/utils.py#L808-L814) and websites understand that it's most likely a script that sends a request. [Check what's your user-agent](https://www.whatismybrowser.com/detect/what-is-my-user-agent/). There's a [how to reduce the chance of being blocked while web scraping blog post](https://serpapi.com/blog/how-to-reduce-chance-of-being-blocked-while-web/) that can get you familiar with basic and more advanced approaches. ## Code Explanation Import libraries: ```python import re, json, time from selenium import webdriver from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager from parsel import Selector ``` | Library | Purpose | | ------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | | [re](https://docs.python.org/3/library/re.html) | to extract parts of the data via regular expression. | | [json](https://docs.python.org/3/library/json.html) | to convert extracted data to a JSON object. | | [time](https://docs.python.org/3/library/time.html) | to work with time in Python. | | [webdriver](https://www.selenium.dev/documentation/webdriver/) | to drive a browser natively, as a user would, either locally or on a remote machine using the Selenium server. | | [Service](https://www.selenium.dev/selenium/docs/api/py/webdriver%5Fchrome/selenium.webdriver.chrome.service.html) | to manage the starting and stopping of the ChromeDriver. | | [Selector](https://parsel.readthedocs.io/en/latest/) | XML/HTML parser that have full [XPath](https://en.wikipedia.org/wiki/XPath) and CSS selectors support. | ### Top-level code environment This code uses the generally accepted rule of using the [\_\_name\_\_ == "\_\_main\_\_"](https://docs.python.org/3/library/%5F%5Fmain%5F%5F.html#) construct: ```python def main(): url = "https://www.youtube.com/watch?v=fbh3OAw3VVQ" result = scroll_page(url) scrape_all_data(result) if __name__ == "__main__": main() ``` This check will only be performed if the user has run this file. If the user imports this file into another, then the check will not work. You can watch the video [Python Tutorial: if **name** \== '**main**'](https://www.youtube.com/watch?v=sugvnHA7ElY&t=1s) for more details. ### Scroll page The function takes a YouTube video page URL and returns a full HTML structure. First, let's understand how YouTube works. YouTube is a dynamic site. The YouTube video page stores a lot of the data you saw in the *what will be scraped* section. Data such as `comments` and `suggested videos` does not load immediately. If the user needs more data, they will simply scroll the page and YouTube will download a small package of data. Accordingly, to get all the data, you need to scroll to the end of the page. In this case, [selenium library](https://www.selenium.dev/documentation/webdriver/) is used, which allows you to simulate user actions in the browser. For `selenium` to work, you need to use [ChromeDriver](https://chromedriver.chromium.org/downloads), which can be downloaded manually or using code. In our case, the second method is used. To control the start and stop of `ChromeDriver`, you need to use [Service](https://www.selenium.dev/selenium/docs/api/py/webdriver%5Fchrome/selenium.webdriver.chrome.service.html) which will install browser binaries under the hood: ```python service = Service(ChromeDriverManager().install()) ``` You should also add `options` to work correctly: ```python options = webdriver.ChromeOptions() options.add_argument("--headless") options.add_argument('--lang=en') options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36") ``` | Chrome options | Explanation | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | \--headless | to run Chrome in headless mode. | | \--lang=en | to set the browser language to English. | | user-agent | to act as a "real" user request from the browser by passing it to [request headers](https://docs.python-requests.org/en/master/user/quickstart/#custom-headers). [Check what's your user-agent](https://www.whatismybrowser.com/detect/what-is-my-user-agent). | Now we can start [webdriver](https://www.selenium.dev/documentation/webdriver/) and pass the url to the [get() method](https://selenium-python.readthedocs.io/api.html#selenium.webdriver.remote.webdriver.WebDriver.get). ```python driver = webdriver.Chrome(service=service, options=options) driver.get(url) ``` The page scrolling algorithm looks like this: 1. Find out the initial page height and write the result to the `old_height` variable. 2. Scroll the page using the script and wait 2 seconds for the data to load. 3. Find out the new page height and write the result to the `new_height` variable. 4. If the variables `new_height` and `old_height` are equal, then we complete the algorithm, otherwise we write the value of the variable `new_height` to the variable `old_height` and return to step 2. Getting the page height and scroll is done by pasting the JavaScript code into the [execute\_script()](https://selenium-python.readthedocs.io/api.html#selenium.webdriver.remote.webdriver.WebDriver.execute%5Fscript) method. ```python old_height = driver.execute_script(""" function getHeight() { return document.querySelector('ytd-app').scrollHeight; } return getHeight(); """) while True: driver.execute_script("window.scrollTo(0, document.querySelector('ytd-app').scrollHeight)") time.sleep(2) new_height = driver.execute_script(""" function getHeight() { return document.querySelector('ytd-app').scrollHeight; } return getHeight(); """) if new_height == old_height: break old_height = new_height ``` Let's use the [Selector](https://parsel.readthedocs.io/en/latest/) from [Parsel library](https://parsel.readthedocs.io/en/latest/), in which we pass the `html` structure with all the data that was received after scrolling the page. This is necessary in order to successfully retrieve data in the next function. After all the operations done, stop the driver: ```python selector = Selector(driver.page_source) # extracting code from HTML driver.quit() ``` The function looks like this: ```python def scroll_page(url): service = Service(ChromeDriverManager().install()) options = webdriver.ChromeOptions() options.add_argument("--headless") options.add_argument('--lang=en') options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36") driver = webdriver.Chrome(service=service, options=options) driver.get(url) old_height = driver.execute_script(""" function getHeight() { return document.querySelector('ytd-app').scrollHeight; } return getHeight(); """) while True: driver.execute_script("window.scrollTo(0, document.querySelector('ytd-app').scrollHeight)") time.sleep(2) new_height = driver.execute_script(""" function getHeight() { return document.querySelector('ytd-app').scrollHeight; } return getHeight(); """) if new_height == old_height: break old_height = new_height selector = Selector(driver.page_source) driver.quit() return selector ``` In the gif below, I demonstrate how this function works: ![blog-youtube-video-page](https://user-images.githubusercontent.com/81998012/183742028-cc3c1721-a851-45d6-a990-f343d49beceb.gif) ### Scrape all data This function takes a full HTML structure and prints all results in json format. It should be noted that there is data that is not displayed on the YouTube video page. To extract them, you need to use [regular expression](https://docs.python.org/3/library/re.html) and search for the `