SerpApi Pagination

Bartek Marmolowski -

Introduction

There are different ways you can extract results from specific APIs. Some include offset pagination, others allow you to access more results by specifying the page, and sometimes, you need to use next-page tokens.

Offset pagination

This pagination is the most popular, as it is used for most Google engines and others like Bing or Baidu. You can specify from which organic results you want the output to be displayed. By default, Google Search API returns ten results per page, so to get the 2nd page, you need to offset your results so that they start from the 11th organic result. Below are a couple of examples of pagination in Google Search API.

Google Search API - 1st page of results
Google Search API - 2nd page of results
Google Search API - 3rd page of results

While Google Search API uses the start parameter to offset your results, other engines might use different parameters, working similarly. You can see a complete list of mentioned parameters below:

Keep in mind that different engines display a different number of results per page, so the offset to access the 2nd page of results from a specific engine needs to be adjusted accordingly. For more information, refer to the documentation links in the list above.

Accessing the following pages directly

Instead of offsetting your results by a specific number, some engines allow you to directly access the 2nd, 3rd, 4th, and other pages by providing the page number. While specific engines allow this form of pagination on top of offset pagination, others restrict it to accessing the following pages this way only. Similarly to the paragraph above, you can refer to a couple of examples of page-like pagination below.

The Home Depot Search API - 1st page of results
The Home Depot Search API - 2nd page of results
The Home Depot Search API - 3rd page of results

Similarly to offset pagination, this method also uses different parameters for specific engines. Below you can find the list with documentation linked to each element.

Using tokens to retrieve more results

The next method to access more results is using the next-page tokens. It is unique because you need to access the previous page to retrieve the token for the following page (e.g.,  you need to access the 1st page to get the token for the 2nd page, etc.). You can access any page anytime in the previously mentioned pagination methods, but the token pagination is different.

You can always find the next_page_token value in the JSON response of your API call for the engine that supports token pagination. Whenever you don't see the next_page_token key at the end of your JSON response, you have reached the last page of the results for your query.

The next_page_token value from the JSON response

Below you can see a couple of examples of the next_page_token usage from Google Maps Reviews API:

Google Maps Reviews - 1st page of results
Google Maps Reviews - 2nd page of results
Google Maps Reviews - 3rd page of results

Even though the key returned in the JSON response of an API call is always called next_page_token, different engines accept the value of this key as various parameters. You can view the full list below:

As you can see, Google Scholar Profiles API uses both the after_author and the before_author parameters. This is the only exception, where you will get the next_page_token and the previous_page_token in your JSON response for all pages of your search query except for the first and the last (the first one will contain only the next_page_token, and the last one only the previous_page_token). Additionally, the sp parameter in YouTube Search API is used for pagination and filtering your results.

Utilizing GET request to access the next page

The last method you can use is an HTTP GET request. You need to construct the SerpApi's link with your parameters, similarly to the example below:

https://serpapi.com/search.json?engine=baidu&q=Coffee&api_key=SECRET_API_KEY
Example of the link used for HTTP GET request

You can use GET requests using tools such as Postman or specific implementations for your favorite programming language. Below you can check a simple example of Baidu Search API pagination with Python's requests library: 

import requests

params = {
  "api_key": "SECRET_API_KEY",
  "engine": "baidu",
  "q": "Coffee"
}

search = requests.get("https://serpapi.com/search.json?", params=params)

while True:
    results = search.json()
    print(results["search_parameters"])
    #printing search parameters for given page

    if 'error' in results:
        print(results["error"])
        break
        # breaking the loop if results return error

    if "next" in results["serpapi_pagination"]:
        search = requests.get(results["serpapi_pagination"]["next"])
        # accessing the following page of results

    else:
        break
        # breaking the loop if there are no more pages to paginate through
Example code in Python to paginate through Baidu Search API results

Increasing the number of results per page

Each engine has a specified default number of results per page. You can bypass this configuration using engine-specific parameters.  For example, if you want to get twenty results per page instead of the default ten in Google Search API, you must use the num=20 parameter in your implementation. You can refer to the examples below to see how to increase the number of results per page:

Google Search API - 100 results returned
Baidu Search API - 50 results returned
Apple App Store API - 200 results returned

Unfortunately, there are limitations regarding how many results you can retrieve from specific APIs. To learn more, please refer to the list below and the documentation pages linked with each position:

Some parameters might not be as reliable as others. For example, Bing's count parameter is only a suggestion for your search query and might not be considered. The same goes for Home Depot's and Walmart's ps parameters, which might sometimes get overridden. This is connected with how each website works, and SerpApi can't get around it.

Conclusion

You can play with different parameters on our Playground to test each API. It allows you to set up your query and get the code for your favorite programming language from the Export To Code option in the top right corner.

If you have any questions or concerns, don't hesitate to get in touch with us at contact@serpapi.com. We will be more than happy to help you!

Join us on Twitter | YouTube