Demystify a SerpApi Request
We use Ruby on Rails for our APIs. Ruby helps us build the best SERP APIs, scrapers for the data scraping community.
Many customers use our SERP APIs because we provide rich data results that are well-documented and fast APIs.
Let's look closer at our API request to see how it works and why it's fast.
Grab an API key
SerpApi offers a free plan for newly created accounts. Head to the sign-up page to register an account and grab your API key. In this example, we will demystify the search results for the query "Coffee" on Google.
We will use curl to make a request to the SerpApi API endpoint. You can use Postman or any other tool to make the request.
curl -s 'https://serpapi.com/search.json?q=Coffee&api_key=YOUR_API_KEY'
If you did this right, you should see a JSON response with search results for the query "Coffee" on Google.
{
"search_metadata": {
"id": "66c3556d2c83d58253c17c91",
"status": "Success",
...
},
"local_map": {
"link": "https://www.google.com/search?sca_esv=2aa9326b59375f61&q=Coffee&npsic=0&rflfq=1&rldoc=1&rllag=33595294,-112238985,1790&tbm=lcl&sa=X&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQtgN6BAgxEAE",
"image": "https://serpapi.com/searches/66c3556d2c83d58253c17c91/images/41036b5cb17bf649a5cfc8c3170af3e0.png",
"gps_coordinates": {
"latitude": 33.595294,
"longitude": -112.238985,
"altitude": 1790
}
},
"local_results": {
"places": [
{
"position": 1,
"rating": 4.9,
"reviews_original": "(243)",
"reviews": 243,
"description": "Dine-in·Takeout·Delivery",
"place_id": "6198093508017199979",
"place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&ludocid=6198093508017199979&q=Coffee",
"lsig": "AB86z5XyEvOdgNOqWlm7WnEu_00_",
"thumbnail": "https://serpapi.com/searches/66c3556d2c83d58253c17c91/images/829ff89a094d741bdd4acf57b62941da61b75216167121825304cfa4877637185efcaef95a48badc.jpeg",
"service_options": {
"dine_in": true,
"takeout": true,
"delivery": true
},
"title": "808 Social AZ",
"type": "Coffee shop",
"address": "8435 W Peoria Ave"
},
...
],
"more_locations_link": "https://www.google.com/search?sca_esv=2aa9326b59375f61&tbs=lf:1,lf_ui:9&tbm=lcl&q=Coffee&rflfq=1&num=10&sa=X&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQjGp6BAg3EAE"
},
"related_questions": [
{
"question": "Is coffee good or bad for health?",
"snippet": "High intake of boiled, unfiltered coffee has been associated with mild increase in cholesterol levels. The bottom line? Your coffee habit is probably fine and may even have some benefits. But if you have side effects from coffee, such as heartburn, nervousness or insomnia, consider cutting back.",
"title": "Coffee and health: What does the research say? - Mayo Clinic",
"link": "https://www.mayoclinic.org/healthy-lifestyle/nutrition-and-healthy-eating/expert-answers/coffee-and-health/faq-20058339#:~:text=High%20intake%20of%20boiled%2C%20unfiltered,or%20insomnia%2C%20consider%20cutting%20back.",
"displayed_link": "https://www.mayoclinic.org › faq-20058339",
"source_logo": "https://serpapi.com/searches/66c3556d2c83d58253c17c91/images/61a81ca273a9f176460399c9adb8925ad9d0ce6fe3eccbdf4890c7eed18538f4.png",
},
...
],
"related_searches": [
{
"block_position": 1,
"query": "Coffee near me",
"link": "https://www.google.com/search?sca_esv=2aa9326b59375f61&q=Coffee+near+me&sa=X&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQ1QJ6BAhjEAE",
"serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&q=Coffee+near+me"
},
...
],
"things_to_know": {
"buttons": [
{
"text": "Caffeine Amount",
"subtitle": "How Much Caffeine In Coffee?",
"title": "Keurig® K-Cup® Pods FAQs & Support",
"link": "https://www.keurig.ca/support/k-cup-pods-packs#:~:text=Each%20K%2DCup%C2%AE%20pod,the%20less%20caffeine%20it%20contains.",
"displayed_link": "https://www.keurig.ca › support › k-cup-pods-packs",
"snippet": "Each K-Cup® pod contains between 75 and 150 mg of caffeine per 250 ml (8 oz) cup. However, this depends on the roasting strength along with other factors. Generally, the darker the coffee is roasted, the less caffeine it contains.",
"snippet_highlighted_words": [
"between 75 and 150 mg of caffeine per 250 ml (8 oz) cup"
],
"answer": "between 75 and 150 mg",
"favicon": "https://serpapi.com/searches/66c3556d2c83d58253c17c91/images/4277885b56aa752b1cc05fee0448d2e2df0c27c769151b13254c24ea277cb395f0a148649ccf5cb4.png",
"source": "keurig.ca",
"search_link": "https://www.google.com/search?sca_esv=2aa9326b59375f61&q=how+much+caffeine+in+coffee&sa=X&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQrooIegQIPBAF",
"serpapi_search_link": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&q=how+much+caffeine+in+coffee"
},
...
]
},
"pagination": {
"current": 1,
"next": "https://www.google.com/search?q=Coffee&sca_esv=2aa9326b59375f61&ei=blXDZvTMErW7kPIP1PCwyQQ&start=10&sa=N&sstk=AagrsuiBDTjyFrMmA3qm4s8Fcr0vraqoHJGX9RhZ_xbhckQS5WBQA4WEDMt9l70d--iRfZHqo-U9L85UQxLjepqJxdWRKX4DCEU08Q&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQ8NMDegQIBxAW",
"other_pages": {
"2": "https://www.google.com/search?q=Coffee&sca_esv=2aa9326b59375f61&ei=blXDZvTMErW7kPIP1PCwyQQ&start=10&sa=N&sstk=AagrsuiBDTjyFrMmA3qm4s8Fcr0vraqoHJGX9RhZ_xbhckQS5WBQA4WEDMt9l70d--iRfZHqo-U9L85UQxLjepqJxdWRKX4DCEU08Q&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQ8tMDegQIBxAE",
...
}
},
"serpapi_pagination": {
"current": 1,
"next_link": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&q=Coffee&start=10",
"next": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&q=Coffee&start=10",
"other_pages": {
"2": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&q=Coffee&start=10",
...
}
}
}
How it works
Users to SerpApi server
We hosted many server instances, about 150 servers at the time of writing this article. Each server runs a Ruby on Rails application that serves the API requests. We use the Puma webserver to handle incoming requests.
You may wonder which server will handle your request and no server is overload. This is the exactly purpose of load balancers. Load balancer will distribute the request to available capable server.
We did many benchmark to configure our server to handle your request with best performance. It deserves a long, detailed posts. We will share more in following blog posts.
Full request lifecycle
How do we return live results from search engines, the only way to achieve this is to do live requests to Search engines. Like your regular searches, we will make requests to search engines, but we are scraping million search engine results pages(SERP) without getting blocked, because we introduced a lot of micro-optimizations from our requests and use "quality" proxies.
Proxies are very important to us, quality proxies reduce response time from Search Engine. Even with quality proxies, sometimes it got bad performance, we have to wait for 10+ seconds to get response from search engine. So how do SerpApi ensure you get your results with seconds?
We come up with a clever solution, a most loved feature from our customer: Ludicrous Speed. We will make multiple requests parallel, once we got response from Search Engines, we cancel all other requests and proceed next step.
Quality and Ludicous Speed are not the only optimized techniques we make SerpApi API super fast.
Once we received response from Search Engine, most of the time, it will be a raw HTML file. Then we parse the HTML file, extract data and return JSON results to you.
We used latest, fastest libaray to speed this process. We also wrote a new Ruby lib nokolexbor (Thanks to lexbor for your amazing project) for a better performance.
Beside these big optimizations, we also did a lot micro optimization to reduce our API response time. If these works sound interesting to you, feel free to send your application!
I hope this blog post will help you feel how we take of a request and build one of the best SERP APIs in the market.
If you have any questions, please feel free to contact me.