SerpApi streamlines the process of scraping search results pages. A common use case is scraping product info from Google, HomeDepot, and/or Walmart.
While still much simpler than manually scraping these sites yourself, getting product details is slightly less straightforward than some of the other data you can scrape with SerpApi.
For example, look at this Google Shopping API search example in the SerpApi Playground:
You can see there is some information available here about each product. But how do we get reviews, ratings, buying options, and sellers?
If you were searching in your browser, you could click on the product to expand it, then click “View product details” to get more information.
But how can we do this with SerpApi, since we need to be able to do it programmatically?
Google Shopping API and Google Product API
If you go back to the Playground and click the dropdown in the top left corner, you will see that in addition to the Google Shopping API, we also have a Google Product API:
However, if you select the Google Product API in the Playground, you will notice that the only way to search is with something called a
If you go back to the Google Shopping API example and click on one of the products to select it, then look over at the selected part of the JSON response, you will notice there is a key named
This is what we need in order to find the item with the Product API:
Home Depot API
The process for scraping product details from Home Depot is almost identical to the one for scraping them from Google.
First we have to use the Home Depot Search API to find the product and get the
Then we can search the Home Depot Product API with the
There are a few differences when scraping product details from the Walmart API. While product results in a Walmart search do have a
product_id, they also have a
us_item_id. We need to use the
us_item_id if we want the results to include Reviews.
First with the Walmart Search API to get the
product_id and the
We can then get the product page with the Product API using the product_id, but there won’t be any reviews:
So if we want the reviews we need to use the
us_item_id. Don't be confused by the name of the parameter, it says
product_id but we can set it's value to the
If you scroll down, you can see there are reviews:
We also have a separate API for getting Walmart Product Reviews. This API also requires the
us_item_id to be supplied as the value of the
We have seen how to get product details from Google, Home Depot, and Walmart using SerpApi.
I hope you found the tutorial informative and easy to follow. If you have any questions, feel free to contact me at email@example.com.
- Google Shopping API
- Google Product API
- Home Depot Search API
- Home Depot Product API
- Walmart Search API
- Walmart Product API
- Walmart Product Reviews API