> ## Content Index > Fetch the complete content index at: https://serpapi.com/blog/llms.txt > Use this file to discover other available public pages before exploring further. # Scrape Google Books in Python - URL: https://serpapi.com/blog/scrape-google-books-in-python/ - Published: 2022-03-16T15:32:34.000Z - Updated: 2023-05-24T20:39:47.000Z - Description: Shows how to scrape title, link, displayed link, snippet, author, pub date, thumbnail, preview, and more editions link. - Author: Dmitriy Zub - Tags: Google Books, Web Scraping, Python ## Intro Currently, we don't have an API that supports extracting data from Google Books. This blog post is to show you how you can do it yourself with the provided DIY solution below while we're working on releasing our proper API. The solution can be used for personal use as it doesn't include the [Legal US Shield](https://serpapi.com/#features) that we offer for our paid [production and above plans](https://serpapi.com/pricing) and has its limitations such as the need to bypass blocks, for example, CAPTCHA. You can check our public roadmap to track the progress for this API: ๐Ÿ—บ๏ธ [\[New API\] Add Google Books API](https://github.com/serpapi/public-roadmap/issues/63) ## What will be scraped ![image](https://user-images.githubusercontent.com/78694043/153621290-26216177-2574-43f5-8676-82237e46b1b1.png) ๐Ÿ“ŒNote: For now, we don't have an API that supports extracting Google Books data. This blog post is to show you how you can do it yourself while we're working on releasing our proper API in the meantime. We'll update you on our Twitter once this API will be released. ## Prerequisites **Basic knowledge scraping with CSS selectors** `CSS` selectors declare which part of the markup a style applies to, thus allowing you to extract data from matching tags and attributes. If you haven't scraped with `CSS` selectors, there's a dedicated blog post of mine about [how to use CSS selectors when web-scraping](https://serpapi.com/blog/web-scraping-with-css-selectors-using-python/) that covers what it is, pros and cons, and why they matter from a web-scraping perspective. **Separate virtual environment** In short, it's a thing that creates an independent set of installed libraries including different Python versions that can coexist with each other at the same system, thus preventing libraries or Python version conflicts. If you haven't worked with a virtual environment before, have a look at the dedicated [Python virtual environments tutorial using Virtualenv and Poetry](https://serpapi.com/blog/python-virtual-environments-using-virtualenv-and-poetry/) blog post of mine to get familiar. ๐Ÿ“ŒNote: This is not a strict requirement for this blog post. **Install libraries**: ```lang-none pip install requests parsel ``` **Reduce the chance of being blocked** There's a chance that a request might be blocked. Have a look at [how to reduce the chance of being blocked while web-scraping](https://serpapi.com/blog/how-to-reduce-chance-of-being-blocked-while-web/), there are eleven methods to bypass blocks from most websites. --- ## Full Code ```python from parsel import Selector import requests, json, re params = { "q": "richard branson", "tbm": "bks", "gl": "us", "hl": "en" } headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36", } html = requests.get("https://www.google.com/search", params=params, headers=headers, timeout=30) selector = Selector(text=html.text) books_results = [] # https://regex101.com/r/mapBs4/1 book_thumbnails = re.findall(r"s=\\'data:image/jpg;base64,(.*?)\\'", str(selector.css("script").getall()), re.DOTALL) for book_thumbnail, book_result in zip(book_thumbnails, selector.css(".Yr5TG")): title = book_result.css(".DKV0Md::text").get() link = book_result.css(".bHexk a::attr(href)").get() displayed_link = book_result.css(".tjvcx::text").get() snippet = book_result.css(".cmlJmd span::text").get() author = book_result.css(".fl span::text").get() author_link = f'https://www.google.com/search{book_result.css(".N96wpd .fl::attr(href)").get()}' date_published = book_result.css(".fl+ span::text").get() preview_link = book_result.css(".R1n8Q a.yKioRe:nth-child(1)::attr(href)").get() more_editions_link = book_result.css(".R1n8Q a.yKioRe:nth-child(2)::attr(href)").get() books_results.append({ "title": title, "link": link, "displayed_link": displayed_link, "snippet": snippet, "author": author, "author_link": author_link, "date_published": date_published, "preview_link": preview_link, "more_editions_link": f"https://www.google.com{more_editions_link}" if more_editions_link is not None else None, "thumbnail": bytes(bytes(book_thumbnail, "ascii").decode("unicode-escape"), "ascii").decode("unicode-escape") }) print(json.dumps(books_results, indent=2)) ``` ### Code explanation Import libraries: ```python from parsel import Selector import requests, json ``` - [parsel](https://parsel.readthedocs.io/en/latest/index.html) is a library to extract and remove data from HTML and XML using XPath and CSS selectors. It's similar to `beautifulsoup4` except it supports full XPath and has its own CSS pseudo-elements support, for example `::text` or `::attr()`. Create search query parameters and request headers: ```python # https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls params = { "q": "richard branson", # search query "tbm": "bks", # book results "gl": "us", # country to search from "hl": "en" # language } # https://requests.readthedocs.io/en/master/user/quickstart/#custom-headers headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36", } ``` - `user-agent` is used to act as a "real" user visit so websites think it's a user, not the bot/script that sends a request. It's the most basic form of avoiding being blocked by a website. Pass query params, request headers to the request and create a `Selector` object: ```python html = requests.get("https://www.google.com/search", params=params, headers=headers, timeout=30) selector = Selector(text=html.text) ``` - [timeout=30](https://docs.python-requests.org/en/master/user/quickstart/#timeouts) tells `requests` to stop waiting for a response after 30 seconds. - `Selector()` is like `BeautifulSoup()` except you get a full XPath support, and [every CSS selector query translates to XPath](https://github.com/scrapy/parsel/blob/f5f73d34ba787ad0c9df25de295de6e196ecd91d/parsel/selector.py#L350-L351) using [cssselect](https://github.com/scrapy/cssselect) package and names it [FunctionalPseudoElement](https://github.com/scrapy/cssselect/blob/f564dfd93358ea55bbd55b75beeb71872a06ed12/cssselect/parser.py#L142). Create a temporary `list` to store the data: ```python books_results = [] ``` Match thumbnails data using a regular expression: ```python # https://regex101.com/r/mapBs4/1 book_thumbnails = re.findall(r"s=\\'data:image/jpg;base64,(.*?)\\'", str(selector.css("script").getall()), re.DOTALL) ``` The reason why we need to parse the data from `