Intro

Search engines are a modern marvel that give us access to a wealth of information at our fingertips. What about harnessing this power to achieve it's full potential, by including automated web search functionality in a software application?

There are about a million and one use cases for this, you can see some examples here:

SerpApi: Use Cases
See how SerpApi can benefit your business. From SEO rankings to news monitoring and machine learning applications, among others.
SerpApi

If you've found your way to this blog, you may already be aware of what SerpApi is and what we do - we scrape search results and return them in an organized JSON format.

However you may also be curious about why this is necessary. Can't you just get the search results directly from Google, Bing, or Yahoo?

We will explore the options available in the current landscape, and circle back on why web scraping is necessary, and why SerpApi is the best option for many developers.

Google

A number of competitors have entered the scene in the past few decades, but Google is still the worlds largest and most prevalent search engine. It's no wonder that many people think of Google first when they think of search results.

It would be helpful to be able to directly access Google Search results programmatically. However, Google does not offer an API that allows searching the web in the way you might expect.

Google Web Search API

It's hard to find good information about Google's early APIs, but it appears they have slowly reduced programmatic access since the mid 2000's. For example, in 2006 they retired their SOAP Search API , replacing it with the more limited JSON Search API.

Google Web Search API was depreciated in 2010 and Google pulled the plug on September 29th, 2014.

Currently, the only option Google provides for searching the web programmatically is the Custom Search API. Google claimed the Custom Search API would provide a "Better Integrated Search Experience" however it was targeted at a very specific use case, as we will see below.

Google Custom Search JSON API

Google's Custom Search JSON API provides search results from a Google Programmable Search Engine in a JSON format. They offer 100 search queries per day for free which may sound like an attractive offer. However, to understand what this offer actually entails, you need to know what a Google Programmable Search Engine is.

The primary purpose of the Google Programmable Search Engine is to allow you to easily create your own search engine for a specific website or collection of websites. When customizing this search engine, there is an option to select "Search the entire web". You can also enable image search, and/or restrict search results to a particular region.

However, even with these options enabled, Google Programmable Search Engines are fundamentally different from the real Google Search Engine.

From Google's FAQ:

Results returned by a Google Programmable Search Engine don't include knowledge graphs, carousels, or any of the other "special" result formats returned in a regular Google Search. What they do return is something like a minimalist version of the Organic Results and Ads.

To see what I mean, let's take a look at this search for "coffee" from a Vancouver, BC, Canada IP address:

With Google:

With Google Programmable Search Engine:

If you compare the two, you can see that even if you skip past the Places and People Also Ask results in the real Google Search and the Ads in the Programmable Search Engine search, the form and rankings of the Organic Results is not the same.

For some use cases, the results in the second example might be fine. But if you want to replicate the experience of performing a real Google Web Search, a Programmable Search Engine instance won't accomplish that. It also won't help you track your SEO rankings, since it doesn't use the same algorithm or process to order results as the "real" Google.

Also, exceeding the 100 free queries a day the price is $5 per 1000 queries with a hard cap of 10k queries per day. If you're hoping to perform a million or more searches per month, the Google Custom Search API won't cover that. There used to be a way to request a larger quota, but Google has currently paused that feature indefinitely.

Other Google APIs

Google also exposes certain other kinds of search data through other APIs - i.e Google Maps and Knowledge Graphs. However these have a confusing and convoluted pricing model, and once again are not comparable to performing a real life search in your browser.

Bing Search API

Unlike Google, Bing does offer a true web search API. Their pricing model is also a bit more straightforward than Google's. They offer a free tier with 1,000 transactions per month, and 9 different paid tiers that include different sets of search types and number of requests per second, with different pay-per-use rates.

However, these are quite expensive, with the cheapest tiers costing $15 per 1,000 searches and only including 1-2 search types. They have one tier that includes all search types and their highest TPS rate of 250 transactions per second - this tier costs $25 per 1,000 searches (with the exception of Bing SpellCheck which is $25 per 25,000 transactions).

Yahoo

Yahoo used to offer something called Yahoo Search BOSS, which included a BOSS JSON Search API. However they discontinued Search BOSS in 2016. They don't currently have anything like a web search API.

DuckDuckGo

DuckDuckGo offers a DuckDuckGo Instant Answers API, however it only provides access to their Instant Answers, it is not a full web search API

Yandex

Yandex offers Yandex.XML, a search API that returns results in XML format. They have strict restrictions on how these results can be used, including requiring a Yandex logo on any page generated using Yandex.XML.

SerpApi

Having explored some of the options available today for accessing search results programmatically, we have seen the limitations of the APIs provided by search giants like Google and Bing. I'm sure there is a use case for which each of the providers mentioned is the most applicable option, however until these companies make search data more directly accessible to developers, web scraping is the only option for many companies.

At SerpApi, we perform live searches, and scrape the results into an organized JSON format without modifying the order or content in any way. This makes SerpApi a good alternative if you want to check your SEO rankings or simulate a real internet search with one of the popular search engines.

We also provide access to results from more search engines than any other service - including Google, Bing, Yahoo, Yandex, Baidu, DuckDuckGo and more.

SerpApi is also a good option if you need to perform a high volume of searches each month. You can get a million or more searches at $2/per 1,000, and we offer monthly plans with up to 100 million searches per month compared to Google's hard-cap of 10,000.

Our lowest tier is the Developer Plan. For $50 per month you get 5,000 searches. That's still $5 per 1,000 searches compared to Bing's $15 - $25 per 1,000 transactions - and you get access to all of the APIs mentioned above. We also offer a scalable pricing model - each increase in volume up to 1 million gives you a better per search rate.

We have faster response times and better success rates than any other service that scrapes search engine results pages - you can see the status of all of our APIs here:

Api Status for All Engines - SerpApi
View SerpApi’s API performance status Live. Monitor search success rates and search response times in real-time.
SerpApiSerpApi

Conclusion

I hope you have found this article informative and easy to follow. If you have any questions, feel free to contact me at ryan@serpapi.com.