Scraping Google Maps with SerpApi - an Overview of the Google Maps API
Intro
SerpApi streamlines the process of scraping search results pages. Today we will take a look inside the powerful Google Maps API. This feature makes it easy to scrape businesses and locations for info, photos, and reviews. You will see that whether you are starting with name of a specific location, a keyword search for a list of related businesses, or a list of IDs you have collected for a location, you can find the data you are looking for using the right query or sequence of queries. This can be a powerful tool for content aggregators, business owners, entrepreneurs, and others. To see this in action, head to the Playground, where you can try out different search queries and parameters.
This post is intended to provide a general overview of how to scrape Google Maps using SerpApi. We will use the Playground to demonstrate the process and make each step clear. We won't delve into any code or implementation details specific to a particular method or programming language. If you are looking for more details about how to do this with or without code, I recommend this tutorial for scraping Google Maps Local Results with Python, or this post about scraping Google Maps into a Google Sheets spreadsheet. You can also view the Related Post links at the bottom of the page.
Google Maps Local Results API
The usual flow for scraping Google Maps begins with a Google Maps Local Results API search. This is because in order to get detailed information about a business or location, we need one of two unique identifiers: place_id
and data_id
. Some users may already have a list of these they have collected using a different method, but most need to get them by searching Google Maps for either a specific business, or a keyword.
You can use the Google Maps Local Results API by making sure the Google Maps engine
is selected, and setting the type
parameter to "Search:
You will also need to specify a q
(query) parameter and an ll
(GPS coordinates) parameter. The Playground will supply some default values for these after you select the engine
and type
. In this example we are using the default paramaters provided by the Playground to search for locations with the keyword "coffee" in New York:
You can see in the example above that a lot of information is available for these businesses just by using the Google Maps Local Results API.
Querying Google Maps Place Results API
If you need more details about a business or location than are available in the Google Local Results, we can use the Google Maps Place Results API.
The easiest way to do this is with the place_id
.
Go back to the Google Maps Local Results API and search for McDonald's.
Let's say the top result is the one we are looking for. Scroll down in the JSON output and find the result in position 1. You can see there is a key named place_id
.
We will now use the place_id
to query the Google Maps Place Results API. For this we can use the regular Google Maps engine
and besides place_id
we don't need to include any other parameters - though it is generally recommended to use all of the available location parameters with queries to SerpApi. You can see in the example below that we have only provided the place_id
:
What if we don't have the place_id
for this location but we do have the data_id
? We can also search Google Maps Place Results with the data_id
but it is a little more complicated. We have to set type=place
and we also need to provide a value for the data
parameter. You might think you can just pass the data_id
here but to create the data
parameter we actually need to use the following format:
!4m5!3m4!1s
+ data_id
+ !8m2!3d
+ latitude
+ !4d
+ longitude
So we will also need the latitude and longitude of the location in order to search for it with the data_id
. If you look again at our Google Maps search for McDonald`s you will see that the GPS coordinates are also returned in the results for each location listed:
So our `data parameter will be !4m5!3m4!1s
+ 0x89c25856329e3e85:0x36266aa626175152
+ !8m2!3d
+ 40.762268999999996
+ !4d
+ -73.9837579
or "!4m5!3m4!1s0x89c25856329e3e85:0x36266aa626175152!8m2!3d40.762268999999996 !4d-73.9837579"
Click here to see it in the Playground.
Google Maps Reviews API
So how do we get reviews for a business? Go back to the Playground and select the Google Maps Reviews engine.
At the top of the window you will see there is a required parameter named data_id
with a value that looks like a string of random letters and numbers.
We need the data_id
in order to get the reviews for a given business. So if we want to get all the reviews for a nearby McDonald's, we first need to go back to the Google Maps Local Results API and search for McDonald's:
Let's say the top result is the one we are looking for. Scroll down in the JSON output and find the result in position 1, then find the data_id
. Just below the place_id
we extracted to use with the Google Maps Place Results API, you can see there is a key named data_id
.
Now we can go back to Google Maps Reviews, and search using the data_id
:
Google Maps Photos API
You can scrape photos and info about photos from any business or location with the Google Maps Photos API and the Google Maps Photo Meta API. Google Maps Photos requires the data_id
of the location - this is the same data_id
we can use to get Reviews, or construct a data
parameter for searching Places. Google Maps Photo Meta also requires a data_id
, but it is the data_id
of the particular photo, not of the location. Don't get confused and try to use the data_id
of a location to search the Google Maps Photo Meta API - it won't work.
The Google Maps Photos API and the Google Maps Photo Meta API also have their own separate engines - google_maps_photos
and google_maps_photo_meta
respectively.
Instead of looking at photos of McDonald's, lets search for Yellowstone Park. First we need to do a regular Google Maps (or Google Maps Local Results) search to get the data_id
for the location:
Now we can search with the Google Maps Photos API:
If we want detailed information about the first photo, we will need to get it's data_id
from the photo_meta_serpapi_link
:
Then we can use the data_id
in a request to the Google Maps Photo Meta API:
Getting the data_id
with a place_id
or Vice Versa
You might find yourself in a situation where you have one kind of ID and you need the other. For example, you might have a list of place_id
s you've collected, but in order to use the Google Maps Reviews API or the Google Maps Photos API you need the data_id
s. This is fortunately very easy to remedy. We've already seen that either a data_id
or a place_id
can be used to query the Google Maps Place Results API.
If you make a request to Google Maps Place Results API you can see that both the data_id
and the place_id
are included in the JSON:
Conclusion
We've seen how to scrape businesses and locations for info, photos, and reviews using SerpApi's Google Maps API.
I hope you found the tutorial informative and easy to follow. If you have any questions, feel free to contact me at ryan@serpapi.com.
Links
Documentation:
- Google Maps API
- Google Maps Local Results API
- Google Maps Place Results API
- Google Maps Reviews API
- Google Maps Photos API
- Google Maps Photo Meta API