Google Local Services API
Baidu Search API
Bing Search API
DuckDuckGo Search API
Ebay Search API
Walmart Search API
The Home Depot Search API
Naver Search API
TypeScript
Ruby
Golang
.Net
C++
What are AI Overviews?
AI overviews are a new feature in Google Search results, which uses generative AI to provide answers to some search queries. These are usually answers to questions with links to references.
Or in their own words:
Google first released this feature back in May (2024), as a part of what was then called the Search Generative Experience, or SGE. They later changed the name to AI Overviews.
AI Overviews are currently only available in English and in select countries, though Google will likely continue to rollout AIO in other countries and languages over time.
You can read more about Google AI Overviews here.
SerpApi Support for AI Overviews
At SerpApi, we specialize in scraping public information from search engine results pages (SERPs), and returning it live in an easy to work with nested JSON format.
We received a high volume of requests to scrape the new AI feature from Google from the time it was first announced. However, until very recently, AI Overviews were only available to Google users logged into a private account. Since our mission is to scrape data from public search results, it's against our policy to scrape anything that requires logging into a private account.
Now that AI Overviews are publicly available without needing to sign in to an account, we're pleased to announce we've developed a solution to parse and return this information in Google Search API responses. You can find the documentation here:
SerpApi
You can also try out an example in our Playground, a useful tool for experimenting with the various APIs we offer:
SerpApiWhy Scrape Google AI Overviews?
There are a number of reasons collecting data from Google AI Overviews might be helpful for your business. For one thing, it's been suggested that AI Overviews may have a higher click-through rate (CTR), so it could be useful to analyze how frequently your site is linked to in AIOs. It could also be useful as part of a general analysis of SEO, or to understand how AIO works, since Google has shared limited information about this. Alternatively, since AIOs provide a succinct summary of answers to complex questions, being able to query for these in real-time could be helpful for developers of applications like chatbots or AI tools that incorporate information from the web.
Why use SerpApi to Scrape AI Overviews?
SerpApi streamlines the process of web-scraping. We take care of proxies and any CAPTCHAs that might be encountered, so that you don't have to worry about your searches being blocked. If you were to implement your own scraper, you'd need to include your own solution for this, which can be a complex process.
We also work to maintain all of our parsers and adapt them to respond to changes on Google's side. This is important, as Google is constantly experimenting with new layouts, new elements, and other changes. By taking care of this for you on our side, we eliminate a lot of time and complexity from your work flow.
How to Scrape AI Overviews
To scrape AI Overviews using SerpApi, you just need to send a query to our Google Search API that is likely to return an AIO result. Asking a question that may have a complex answer is the best way to get an AIO result.
Here is an example of sending the query "how to brew good coffee" to SerpApi using our Python library:
from serpapi import GoogleSearch
params = {
"q": "how to brew good coffee",
"api_key": "YOUR_SECRET_API_KEY"
}
search = GoogleSearch(params)
results = search.get_dict()
ai_overview = results["ai_overview"]The following is returned:
"ai_overview": {
"text_blocks": [
{
"type": "paragraph",
"snippet": "Here are some tips for brewing good coffee:",
"reference_indexes": [
1
]
},
{
"type": "list",
"list": [
{
"title": "Use a medium grind",
"snippet": "A medium-ground coffee is a good place to start. Finer grinds extract more flavor, while coarser grinds produce more caffeine.",
"reference_indexes": [
1,
2
]
},
{
"title": "Use filtered water",
"snippet": "The quality of the water you use can significantly impact the taste of your coffee.",
"reference_indexes": [
1
]
},
{
"title": "Grind right before brewing",
"snippet": "Grinding your beans closer to the time of brewing helps ensure a fresher taste.",
"reference_indexes": [
1
]
},
{
"title": "Use the right temperature",
"snippet": "The ideal water temperature for brewing coffee is between 195–205°F, which is just below boiling.",
"reference_indexes": [
5
]
},
{
"title": "Be patient",
"snippet": "Don't set up your coffee too early, as the flavor will be better if you wait until morning.",
"reference_indexes": [
1
]
},
{
"title": "Turn off the heat",
"snippet": "Turn off the heat when you're done brewing, as the coffee will taste worse the longer it stays on the burner.",
"reference_indexes": [
1
]
},
{
"title": "Clean your machine",
"snippet": "Clean your coffee machine once a month by brewing a full pot of a mixture of one part vinegar to two parts water, followed by a full pot of hot water.",
"reference_indexes": [
1
]
},
{
"title": "Consider the brewing method",
"snippet": "Different brewing methods, such as French press, pour-over, cold brew, Aeropress, and Moka pot, each have their own unique qualities.",
"reference_indexes": [
3,
4,
6,
8,
9
]
}
]
}
],
"references": [
{
"title": "11 Tips for Brewing Better Coffee at Home | Caffe Luxxe",
"link": "https://www.caffeluxxe.com/blogs/news/11-tips-for-brewing-better-coffee-at-home",
"snippet": "11 Tips for Brewing Better Coffee at Home * Use A Scale. ... * Use The Right Amount of Coffee. ... * Use A Burr Grinder. ... * Gr...",
"source": "Caffe Luxxe",
"index": 0
},
{
"title": "How to Brew Good Drip Coffee",
"link": "https://primeroastcoffee.com/about-prime-roast-coffee/emilys-coffee-brewing-guides/how-to-brew-good-drip-coffee/",
"snippet": "* Start with a good medium-grind coffee. Buy exceptionally good coffee. Use a medium grind. Pro tip: The closer you grind your be...",
"source": "Prime Roast Coffee",
"index": 1
},
{
"title": "How to Make the Perfect Cup of Coffee",
"link": "https://atlascoffeeclub.com/pages/how-to-make-the-perfect-cup-of-coffee#:~:text=A%20finer%20grind%20%E2%80%93%20which%20means%20smaller,always%20want%20to%20aim%20for%20a%20finer",
"snippet": "A finer grind – which means smaller particles with a larger surface area to slow the movement of water -- will extract more of a c...",
"source": "Atlas Coffee Club",
"index": 2
},
{
"title": "6 Popular Methods for Brewing Coffee at Home",
"link": "https://treescoffee.com/blog/2016/04/6-popular-methods-for-brewing-coffee-at-home/#:~:text=Percolate:%20Stovetop%20Moka%20Pot.%20Originally%20patented%20in,coffee%20then%20sits%20in%20the%20higher%20chamber.",
"snippet": "Percolate: Stovetop Moka Pot. Originally patented in 1933 by Alfonso Bialetti, stovetop style coffee makers use steam pressure fro...",
"source": "Trees Organic Coffee",
"index": 3
},
{
"title": "Why You Should Drink Cold Brew Coffee",
"link": "https://stonestreetcoffee.com/blogs/brooklyn-coffee-academy/why-you-should-drink-cold-brew-coffee#:~:text=Allowing%20stale%20coffee%20beans%20to%20steep%20in,while%20the%20bitter%20taste%20takes%20a%20backseat.",
"snippet": "Allowing stale coffee beans to steep in cold water for up to 24 hours filters out the accumulated acidity giving your coffee beans...",
"source": "Stone Street Coffee",
"index": 4
},
{
"title": "How To Make The Perfect Cup of Coffee At Home",
"link": "https://eldoradocoffee.com/blogs/news/perfect-coffee-home",
"snippet": "Jan 23, 2020 — Boiling is the key to how to make good coffee at home Do you boil your water? Good! Boiling water helps ensure the maj...",
"source": "Eldorado Coffee Roasters",
"index": 5
},
{
"title": "Best Ways to Brew Coffee While Van Camping",
"link": "https://freedomvans.co/community/best-ways-to-brew-coffee-while-van-camping/#:~:text=Aeropress:%20Aeropress%20is%20a%20favorite%20because%20it,fresh%2C%20just%20before%20you're%20going%20to%20brew.",
"snippet": "Nov 2, 2020 — Aeropress: Aeropress is a favorite because it makes one flavorful, strong cup of coffee. Aeropress tends to pair well w...",
"source": "Freedom Vans",
"index": 6
},
{
"title": "How to Brew Coffee - 7 Popular Methods",
"link": "https://levista.in/blogs/news/how-to-brew-coffee#:~:text=Steeping%20the%20grounds%20for%20this%20amount%20of,preventing%20you%20from%20getting%20a%20bitter%20cup.",
"snippet": "Dec 23, 2022 — Steeping the grounds for this amount of time extracts the flavours and sugars. With this method of brewing coffee, you...",
"source": "levista.in",
"index": 7
},
{
"title": "Pour-Over Coffee: How to Make the Perfect Cup",
"link": "https://unocasa.com/blogs/tips/pour-over-coffee#:~:text=If%20you're%20looking%20for%20the%20best%20method,the%20excitement%20of%20creating%20the%20perfect%20brew.",
"snippet": "If you're looking for the best method for brewing coffee, pour-over has to be the way to go! The precision and attention to detail...",
"source": "Uno Casa",
"index": 8
},
{
"title": "ALL ABOUT MANUAL COFFEE BREWING METHODS – Coffee Hero",
"link": "https://coffeehero.com.au/blogs/news/all-about-manual-coffee-brewing-methods",
"snippet": "Coffee lovers consider the French press one of the best and easiest methods of brewing coffee. The brewer gives more superior flav...",
"source": "Coffee Hero",
"index": 9
}
]
},SerpApi supports libraries for the top programming languages, including Ruby, Python, JavaScript, PHP, and Java.
If you are not a programmer, you can also see what this looks like using our Playground:
SerpApiOr, you can try one of our no-code options if you want to automate scraping AIO results without writing any code:
Hilman Ramadhan
Variations in the Format of the AIO Result
Most AIO results contain a paragraph and/or list, and possibly a title and/or thumbnail.
However, sometimes the response is a list with thumbnails, a nested list, or it contains sections. SerpApi is able to parse these alternative layouts as well. You can learn more and see some examples of these on the documentation page.
Conclusion
We've covered Google AI Overviews, and how you can scrape them using SerpApi. Hopefully you found this article informative and easy to understand. Feel free to reach out to contact@serpapi.com anytime if you have any questions or concerns.