Demystify a SerpApi Request
We use Ruby on Rails for our APIs. Ruby helps us build the best SERP APIs, scrapers for the data scraping community.
Many customers use our SERP APIs because we provide rich data results that are well-documented as well as fast APIs.
Let's take a closer look at our API request to see how it works and why it's so fast.
Grab an API key
SerpApi offers a free plan for newly created accounts. Head to the sign-up page to register an account and grab your API key. In this example, we will demystify the search results for the query "Coffee" on Google.
We will use curl to make a request to the SerpApi API endpoint. You can use Postman or any other tool to make the request.
curl -s 'https://serpapi.com/search.json?q=Coffee&api_key=YOUR_API_KEY'
If you did this right, you should see a JSON response with search results for the query "Coffee" on Google.
{
"search_metadata": {
"id": "66c3556d2c83d58253c17c91",
"status": "Success",
...
},
"local_map": {
"link": "https://www.google.com/search?sca_esv=2aa9326b59375f61&q=Coffee&npsic=0&rflfq=1&rldoc=1&rllag=33595294,-112238985,1790&tbm=lcl&sa=X&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQtgN6BAgxEAE",
"image": "https://serpapi.com/searches/66c3556d2c83d58253c17c91/images/41036b5cb17bf649a5cfc8c3170af3e0.png",
"gps_coordinates": {
"latitude": 33.595294,
"longitude": -112.238985,
"altitude": 1790
}
},
"local_results": {
"places": [
{
"position": 1,
"rating": 4.9,
"reviews_original": "(243)",
"reviews": 243,
"description": "Dine-in·Takeout·Delivery",
"place_id": "6198093508017199979",
"place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&ludocid=6198093508017199979&q=Coffee",
"lsig": "AB86z5XyEvOdgNOqWlm7WnEu_00_",
"thumbnail": "https://serpapi.com/searches/66c3556d2c83d58253c17c91/images/829ff89a094d741bdd4acf57b62941da61b75216167121825304cfa4877637185efcaef95a48badc.jpeg",
"service_options": {
"dine_in": true,
"takeout": true,
"delivery": true
},
"title": "808 Social AZ",
"type": "Coffee shop",
"address": "8435 W Peoria Ave"
},
...
],
"more_locations_link": "https://www.google.com/search?sca_esv=2aa9326b59375f61&tbs=lf:1,lf_ui:9&tbm=lcl&q=Coffee&rflfq=1&num=10&sa=X&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQjGp6BAg3EAE"
},
"related_questions": [
{
"question": "Is coffee good or bad for health?",
"snippet": "High intake of boiled, unfiltered coffee has been associated with mild increase in cholesterol levels. The bottom line? Your coffee habit is probably fine and may even have some benefits. But if you have side effects from coffee, such as heartburn, nervousness or insomnia, consider cutting back.",
"title": "Coffee and health: What does the research say? - Mayo Clinic",
"link": "https://www.mayoclinic.org/healthy-lifestyle/nutrition-and-healthy-eating/expert-answers/coffee-and-health/faq-20058339#:~:text=High%20intake%20of%20boiled%2C%20unfiltered,or%20insomnia%2C%20consider%20cutting%20back.",
"displayed_link": "https://www.mayoclinic.org › faq-20058339",
"source_logo": "https://serpapi.com/searches/66c3556d2c83d58253c17c91/images/61a81ca273a9f176460399c9adb8925ad9d0ce6fe3eccbdf4890c7eed18538f4.png",
},
...
],
"related_searches": [
{
"block_position": 1,
"query": "Coffee near me",
"link": "https://www.google.com/search?sca_esv=2aa9326b59375f61&q=Coffee+near+me&sa=X&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQ1QJ6BAhjEAE",
"serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&q=Coffee+near+me"
},
...
],
"things_to_know": {
"buttons": [
{
"text": "Caffeine Amount",
"subtitle": "How Much Caffeine In Coffee?",
"title": "Keurig® K-Cup® Pods FAQs & Support",
"link": "https://www.keurig.ca/support/k-cup-pods-packs#:~:text=Each%20K%2DCup%C2%AE%20pod,the%20less%20caffeine%20it%20contains.",
"displayed_link": "https://www.keurig.ca › support › k-cup-pods-packs",
"snippet": "Each K-Cup® pod contains between 75 and 150 mg of caffeine per 250 ml (8 oz) cup. However, this depends on the roasting strength along with other factors. Generally, the darker the coffee is roasted, the less caffeine it contains.",
"snippet_highlighted_words": [
"between 75 and 150 mg of caffeine per 250 ml (8 oz) cup"
],
"answer": "between 75 and 150 mg",
"favicon": "https://serpapi.com/searches/66c3556d2c83d58253c17c91/images/4277885b56aa752b1cc05fee0448d2e2df0c27c769151b13254c24ea277cb395f0a148649ccf5cb4.png",
"source": "keurig.ca",
"search_link": "https://www.google.com/search?sca_esv=2aa9326b59375f61&q=how+much+caffeine+in+coffee&sa=X&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQrooIegQIPBAF",
"serpapi_search_link": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&q=how+much+caffeine+in+coffee"
},
...
]
},
"pagination": {
"current": 1,
"next": "https://www.google.com/search?q=Coffee&sca_esv=2aa9326b59375f61&ei=blXDZvTMErW7kPIP1PCwyQQ&start=10&sa=N&sstk=AagrsuiBDTjyFrMmA3qm4s8Fcr0vraqoHJGX9RhZ_xbhckQS5WBQA4WEDMt9l70d--iRfZHqo-U9L85UQxLjepqJxdWRKX4DCEU08Q&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQ8NMDegQIBxAW",
"other_pages": {
"2": "https://www.google.com/search?q=Coffee&sca_esv=2aa9326b59375f61&ei=blXDZvTMErW7kPIP1PCwyQQ&start=10&sa=N&sstk=AagrsuiBDTjyFrMmA3qm4s8Fcr0vraqoHJGX9RhZ_xbhckQS5WBQA4WEDMt9l70d--iRfZHqo-U9L85UQxLjepqJxdWRKX4DCEU08Q&ved=2ahUKEwj0u8nyoIGIAxW1HUQIHVQ4LEkQ8tMDegQIBxAE",
...
}
},
"serpapi_pagination": {
"current": 1,
"next_link": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&q=Coffee&start=10",
"next": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&q=Coffee&start=10",
"other_pages": {
"2": "https://serpapi.com/search.json?device=desktop&engine=google&google_domain=google.com&q=Coffee&start=10",
...
}
}
}
How it works
Users to SerpApi server
We hosted many server instances, about 300 servers at the time of writing this article. Each server runs a Ruby on Rails application that serves the API requests. We use the Puma web server to handle incoming requests.
You may wonder which server will handle your request and no server is overloaded. This is the exact purpose of load balancers. Load balancers will distribute the request to available capable servers.
We did many benchmarks to configure our servers to handle your requests with the best performance. It deserves a long, detailed post. We will share more in future blog posts.
Full request lifecycle
How do we return live results from search engines? The only way to achieve this is to make live requests to Search engines. Like your regular searches, we will make requests to search engines, but we are scraping a million search engine results pages(SERP) without getting blocked, because we introduced a lot of micro-optimizations from our requests and use "quality" proxies.
Proxies are very important to us, quality proxies reduce response time from Search Engines. Even with quality proxies, sometimes it has bad performance, and we have to wait for 10+ seconds to get a response from search engine. So how does SerpApi ensure you get your results within seconds?
We came up with a clever solution, one of our most-loved features from our customers: Ludicrous Speed. We will make multiple parallel requests, once we get a response from a Search Engine, we cancel all other requests and proceed to the next step.
Quality proxies and Ludicrous Speed are not the only optimized techniques we use to make SerpApi APIs super fast.
Once we have received a response from a Search Engine, most of the time, it will be a raw HTML file. Then we parse the HTML file, extract data and return JSON results to you.
We used the latest, fastest library to speed up this process. We also wrote a new Ruby lib nokolexbor (Thanks to lexbor for your amazing project) for even better performance.
Besides these big optimizations, we also do a lot of micro optimization to reduce our API response time. If this work sound interesting to you, feel free to send us your application!
I hope this blog post will help you see how we take care of a request and build one of the best SERP APIs in the market.
If you have any questions, please feel free to contact me.