> ## Content Index > Fetch the complete content index at: https://serpapi.com/blog/llms.txt > Use this file to discover other available public pages before exploring further. # Scrape Google Product Online Sellers with Python - URL: https://serpapi.com/blog/scrape-google-product-online-sellers-with-python/ - Published: 2022-11-29T11:00:00.000Z - Updated: 2026-01-28T07:03:15.000Z - Description: This blog post is a step-by-step tutorial about scraping Google Product Online Sellers using Python. - Author: Artur Chukhrai - Tags: Google Product, Web Scraping, Python โ€ผ๏ธ ****\[UPDATE\]** Google has shut down the Google Product service. As a result, our Google Product API will now return an error indicating that the Google Product service has been shut down. The [Google Immersive Product API](https://serpapi.com/google-immersive-product-api) contains the same information previously available in the Google Product service. ## What will be scraped ![wwbs-google-online-sellers](https://user-images.githubusercontent.com/81998012/203135852-4f385688-2e0f-4cdb-ba36-e4006e32e203.png) ๐Ÿ“ŒNote: In this image, I demonstrate that the data will be received with pagination. Therefore, I only show 5 sellers, and not all, as the image could take up a lot of space. ## Using Google Online Sellers API from SerpApi This section is to show the comparison between the DIY solution and our solution. The main difference is that it's a quicker approach. [Google Online Sellers API](https://serpapi.com/online-sellers) will bypass blocks from search engines and you don't have to create the parser from scratch and maintain it. First, we need to install [google-search-results](https://pypi.org/project/google-search-results/): ```lang-none pip install google-search-results ``` Import the necessary libraries for work: ```python from serpapi import GoogleSearch import json ``` Next, we write the necessary parameters for making a request: ```python params = { 'api_key': '...', # https://serpapi.com/manage-api-key 'engine': 'google_product', # SerpApi search engine 'product_id': '14019378181107046593', # product id 'offers': True, # more offers, could be also set as '1` which is the same as True 'hl': 'en', # language 'gl': 'us' # country of the search, US -> USA } ``` We then create a `search` object where the data is retrieved from the SerpApi backend. In the `results` dictionary we get data from JSON: ```python search = GoogleSearch(params) # where data extraction happens on the SerpApi backend results = search.get_dict() # JSON -> Python dict ``` Retrieving the data is quite simple, we just need to access the `'sellers_results'` key and then the `'online_sellers'` key: ```python online_sellers = results['sellers_results']['online_sellers'] ``` After reviewing the [playground](https://serpapi.com/playground?engine=google%5Fproduct&product%5Fid=14019378181107046593&gl=us&hl=en&offers=1&device=desktop), you will be able to understand which keys you can turn to into this JSON structure. Example code to integrate: ```python from serpapi import GoogleSearch import json params = { 'api_key': '...', # https://serpapi.com/manage-api-key 'engine': 'google_product', # SerpApi search engine 'product_id': '14019378181107046593', # product id 'offers': True, # more offers, could be also set as '1` which is the same as True 'hl': 'en', # language 'gl': 'us' # country of the search, US -> USA } search = GoogleSearch(params) # where data extraction happens on the backend results = search.get_dict() # JSON -> Python dict online_sellers = results['sellers_results']['online_sellers'] print(json.dumps(online_sellers, indent=2, ensure_ascii=False)) ``` Output: ```json [ { "position": 1, "name": "Best Buy", "link": "https://www.google.com/url?q=https://www.bestbuy.com/site/steelseries-aerox-3-2022-edition-lightweight-wired-optical-gaming-mouse-onyx/6485231.p%3FskuId%3D6485231%26ref%3DNS%26loc%3D101&sa=U&ved=0ahUKEwiYt4fxyb_7AhXGFlkFHQZoCLMQ2ykIJA&usg=AOvVaw198AdAmbpUT5YEupYrp_iH", "base_price": "$34.99", "additional_price": { "shipping": "$0.00", "tax": "$3.02" }, "total_price": "$38.01" }, ... other sellers { "position": 38, "name": "Network Hardwares", "link": "https://www.google.com/url?q=https://www.networkhardwares.com/products/aerox-3-wireless-2022-edition-62611%3Fcurrency%3DUSD%26variant%3D41025510441165%26utm_medium%3Dcpc%26utm_source%3Dgoogle%26utm_campaign%3DGoogle%2520Shopping%26srsltid%3DAYJSbAdn6Cgm7HKsOdgiZ1_T8TK8NyOtSJpq2EC5meylVz982o4QDNcuTfA&sa=U&ved=0ahUKEwiYt4fxyb_7AhXGFlkFHQZoCLMQ2ykI5wE&usg=AOvVaw18MAXohnYThkG5Ip4Igqx-", "base_price": "$64.51", "additional_price": { "shipping": "$0.00", "tax": "$5.56" }, "total_price": "$70.07" } ] ``` ## DIY Code If you don't need explanation, have a look at [full code example in the online IDE](https://replit.com/@chukhraiartur/blog-google-product-online-sellers#main.py). ```python import requests, json from parsel import Selector def get_online_sellers_results(url, headers): data = [] while True: html = requests.get(url, headers=headers) selector = Selector(html.text) for result in selector.css('.sh-osd__offer-row'): name = result.css('.kjM2Bf::text, .b5ycib::text').get() link = 'https://www.google.com' + result.css('.b5ycib::attr(href)').get() if result.css('.b5ycib') else None base_price = result.css('.fObmGc::text').get() shipping = result.css('.SuutWb tr:nth-child(2) td:nth-child(2)::text').get() tax = result.css('.SuutWb tr:nth-child(3) td:nth-child(2)::text').get() total_price = result.css('.drzWO::text').get() data.append({ 'name': name, 'link': link, 'base_price': base_price, 'additional_price': { 'shipping': shipping, 'tax': tax }, 'total_price': total_price }) if 'Next' in selector.css('.R9e18b .internal-link::text').get(): url = 'https://www.google.com' + selector.css('.R9e18b .internal-link::attr(data-url)').get() else: break return data def main(): # https://docs.python-requests.org/en/master/user/quickstart/#custom-headers headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36' } URL = 'https://www.google.com/shopping/product/14019378181107046593/offers?hl=en&gl=us' online_sellers = get_online_sellers_results(URL, headers) print(json.dumps(online_sellers, indent=2, ensure_ascii=False)) if __name__ == "__main__": main() ``` ## Preparation **Install libraries**: ```lang-none pip install requests parsel ``` **Reduce the chance of being blocked** Make sure you're using [request headers](https://docs.python-requests.org/en/master/user/quickstart/#custom-headers) [user-agent](https://developer.mozilla.org/en-US/docs/Glossary/User%5Fagent) to act as a "real" user visit. Because default `requests` `user-agent` is [python-requests](https://github.com/psf/requests/blob/589c4547338b592b1fb77c65663d8aa6fbb7e38b/requests/utils.py#L808-L814) and websites understand that it's most likely a script that sends a request. [Check what's your user-agent](https://www.whatismybrowser.com/detect/what-is-my-user-agent/). There's a [how to reduce the chance of being blocked while web scraping blog post](https://serpapi.com/blog/how-to-reduce-chance-of-being-blocked-while-web/) that can get you familiar with basic and more advanced approaches. ## Code Explanation Import libraries: ```python import requests, json from parsel import Selector ``` | Library | Purpose | | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | [requests](https://requests.readthedocs.io/en/latest/user/quickstart/) | to make a request to the website. | | [json](https://docs.python.org/3/library/json.html) | to convert extracted data to a JSON object. | | [Selector](https://parsel.readthedocs.io/en/latest/) | XML/HTML parser that have full [XPath](https://en.wikipedia.org/wiki/XPath) and CSS selectors support. | At the beginning of the `main()` function, the `headers` and `URL` are defined. This data is then passed to the `get_online_sellers_results(URL, headers)` function to form a request and extract information. The `online_sellers` list contains the received data that this function returns. At the end of the function, the data is output in JSON format: ```python def main(): # https://docs.python-requests.org/en/master/user/quickstart/#custom-headers headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36' } URL = 'https://www.google.com/shopping/product/14019378181107046593/offers?hl=en&gl=us' online_sellers = get_online_sellers_results(URL, headers) print(json.dumps(online_sellers, indent=2, ensure_ascii=False)) ``` This code uses the generally accepted rule of using the [\_\_name\_\_ == "\_\_main\_\_"](https://docs.python.org/3/library/%5F%5Fmain%5F%5F.html#) construct: ```python if __name__ == "__main__": main() ``` This check will only be performed if the user has run this file. If the user imports this file into another, then the check will not work. You can watch the video [Python Tutorial: if **name** \== '**main**'](https://www.youtube.com/watch?v=sugvnHA7ElY&t=1s) for more details. Let's take a look at the `get_online_sellers_results(url, headers)` function mentioned earlier. This function takes `url` and `headers` parameters to create a request. At the beginning of the function, the `data` list in which the data will be stored is defined: ```python def get_online_sellers_results(url, headers): data = [] ``` Now we need to parse the HTML from the [Parsel](https://parsel.readthedocs.io/en/latest/) package, into which we pass the `HTML` structure that was received after the request. Up to 20 sellers fit on one page. If there are more than 20 of them, then a page with the remaining sellers is added. To scrape a Google Product Online Sellers with pagination, you need to check for the presence of the **Next** button. While the **Next** button exists, you need to fetch the `url` for the next page in order to access it. If the **Next** button is not present, then you need to `break` the while loop: ```python while True: html = requests.get(url, headers=headers) selector = Selector(html.text) # data extraction from current page will be here if 'Next' in selector.css('.R9e18b .internal-link::text').get(): url = 'https://www.google.com' + selector.css('.R9e18b .internal-link::attr(data-url)').get() else: break ``` To retrieve data, you first need to find the `.sh-osd__offer-row` selector that is responsible for each seller and iterate over it: ```python for result in selector.css('.sh-osd__offer-row'): # data extraction from each seller will be here ``` Data such as `name`, `base_price`, `shipping`, `tax` and `total_price` are retrieved for each seller. I want to draw your attention to the fact that not every seller has a `link`, so a [ternary expression](https://docs.python.org/3/reference/expressions.html#conditional-expressions) is used when extracting: ```python name = result.css('.kjM2Bf::text, .b5ycib::text').get() link = 'https://www.google.com' + result.css('.b5ycib::attr(href)').get() if result.css('.b5ycib') else None base_price = result.css('.fObmGc::text').get() shipping = result.css('.SuutWb tr:nth-child(2) td:nth-child(2)::text').get() tax = result.css('.SuutWb tr:nth-child(3) td:nth-child(2)::text').get() total_price = result.css('.drzWO::text').get() ``` | Code | Explanation | | ----------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | | [css()](https://parsel.readthedocs.io/en/latest/usage.html#using-selectors) | to access elements by the passed selector. | | [::text or ::attr()](https://github.com/scrapy/parsel/blob/90397dcd0b2c1cbb91e44f65c50f9e11628ba028/parsel/csstranslator.py#L48-L51) | to extract textual or attribute data from the node. | | [get()](https://parsel.readthedocs.io/en/latest/usage.html#usage) | to actually extract the textual data. | After extracting all data about the seller, a dictionary with this data is appended to the `data` list: ```python data.append({ 'name': name, 'link': link, 'base_price': base_price, 'additional_price': { 'shipping': shipping, 'tax': tax }, 'total_price': total_price }) ``` At the end of the function, the `data` list is returned. ```python return data ``` Output: ```json [ { "name": "Best Buy", "link": "https://www.google.com/url?q=https://www.bestbuy.com/site/steelseries-aerox-3-2022-edition-lightweight-wired-optical-gaming-mouse-onyx/6485231.p%3FskuId%3D6485231%26ref%3DNS%26loc%3D101&sa=U&ved=0ahUKEwiSuKKm1r_7AhWESDABHQvhDGwQ2ykIJA&usg=AOvVaw37TQlxlXfUf7Aow3-oj3Wr", "base_price": "$34.99", "additional_price": { "shipping": "$0.00", "tax": "$3.11" }, "total_price": "$38.10" }, ... other sellers { "name": "Network Hardwares", "link": "https://www.google.com/url?q=https://www.networkhardwares.com/products/aerox-3-wireless-2022-edition-62611%3Fcurrency%3DUSD%26variant%3D41025510441165%26utm_medium%3Dcpc%26utm_source%3Dgoogle%26utm_campaign%3DGoogle%2520Shopping%26srsltid%3DAYJSbAeM3Wi-nx6CPNXcQIZqlFcEv3uyBEgwTXa36ijEua1hx_LNmAm5EiM&sa=U&ved=0ahUKEwiSuKKm1r_7AhWESDABHQvhDGwQ2ykImgE&usg=AOvVaw1rOVOsiroUgnyyTT2JBN61", "base_price": "$64.51", "additional_price": { "shipping": "$0.00", "tax": "$5.73" }, "total_price": "$70.24" } ] ``` ## Links - [Code in the online IDE](https://replit.com/@chukhraiartur/blog-google-product-online-sellers#main.py) - [Google Online Sellers API](https://serpapi.com/online-sellers) Join us on [Twitter](https://twitter.com/serp%5Fapi) | [YouTube](https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg%5Fmg) Add a [Feature Request](https://github.com/serpapi/public-roadmap/issues)๐Ÿ’ซ or a [Bug](https://github.com/serpapi/public-roadmap/issues)๐Ÿž