> ## Content Index
> Fetch the complete content index at: https://serpapi.com/blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# SerpApi Async Requests with Pagination using Python
- URL: https://serpapi.com/blog/serpapi-async-requests-with-pagination-using-python/
- Published: 2022-11-17T11:48:52.000Z
- Updated: 2023-03-28T20:16:17.000Z
- Description: How to make Async requests with SerpApi's pagination, how to use Search Archive API and Queue, and how much it faster than Sync requests.
- Author: Dmitriy Zub
- Tags: Performance, Python

## Intro

In the previous [async requests with SerpApi and Python](https://serpapi.com/blog/making-async-requests-with-serpapi/) blog post we've covered how to make async requests and how they work. In this continuation blog post, we'll cover how to add pagination to async requests.

📌Note: This blog post does not cover multithreading or [asyncio](https://docs.python.org/3/library/asyncio.html). The topic will be covered in the next blog post.

---

**Subject of test**: [YouTube Search Engine Results API](https://serpapi.com/youtube-search-api).

**Test includes**: 50 search queries, pagination per each query, and data extraction. Sync and Async used different search queries.

**Where test happend**: using [Replit](https://replit.com/)'s [Boosted](https://blog.replit.com/boosts) repls.

**What hardware used**: 4 vCPUs and 4 GB of RAM on Ubuntu 20.04.2 LTS.

## Time Comparison

The change in total elapsed time is a whopping 434% change 😵😍

Time was recorded using [$ time python <file.py>](https://stackoverflow.com/a/1557577/15164646):

![image](https://user-images.githubusercontent.com/78694043/201845352-e8346864-64f6-4265-9e60-f112a20afb43.png)

| Type    | Sync requests pagination | Async requests pagination | % difference       |
| ------- | ------------------------ | ------------------------- | ------------------ |
| user    | 135.98s                  | 149.98s                   | \-90.67% decrease  |
| system  | 20.43s                   | 13.43s                    | \-152.12% decrease |
| elapsed | 2h 36m 25s               | 36m 2s                    | +434.09% increase  |

## Sync Pagination

You can [check the code example in the online IDE](https://replit.com/@serpapi/SerpApi-YouTube-Search-Sync-with-Pagination#main.py):

```python
from serpapi import YoutubeSearch
from urllib.parse import (parse_qsl, urlsplit)
import json

# shortened for example
queries = [
    'tent',
    'friendly',
    'ripe',
    'helpful',
    'analyze',
    'pack'
]

data = []

for query in queries:
    params = {
        'api_key': '...',                 # https://serpapi.com/manage-api-key
        'engine': 'youtube',              # search engine
        'device': 'desktop',              # device type
        'search_query': query,            # search query
    }

    search = YoutubeSearch(params)        # where data extraction happens

    while True:
        results = search.get_json()       # JSON -> Python dict

        if 'error' in results:
            print(results['error'])
            break

        for result in results.get('video_results', []):
            data.append({
                'title': result.get('title'),
                'link': result.get('link'),
                'channel': result.get('channel').get('name'),
            })
        
        if 'next' in results.get('serpapi_pagination', {}):
            search.params_dict.update(dict(parse_qsl(urlsplit(results.get('serpapi_pagination', {}).get('next')).query)))
        else:
            break

# print(json.dumps(data, indent=2, ensure_ascii=False))

```

### Sync Pagination Explanation

Import libraries:

```python
from serpapi import YoutubeSearch
from urllib.parse import (parse_qsl, urlsplit) # for pagination
import json

```

Create a [list](https://www.w3schools.com/python/python%5Flists.asp) of search queries we want to search:

```python
queries = [
    'burly',
    'silk',
    'monkey',
    'abortive',
    'hot'
]

```

(optional) Create a temporary `list` that will store extracted data:

```python
data = []

```

Add a `for` loop to iterate over all `queries`, create [SerpApi YouTube search parameters](https://serpapi.com/youtube-search-api#api-parameters), and pass them to `YoutubeSearch` which will make a request to SerpApi. Transform the return JSON to Python `dict` via [get\_dict()](https://github.com/serpapi/google-search-results-python/blob/56447f2ac39f202fa663233c87fa7c7b9ca1e6b2/serpapi/serp%5Fapi%5Fclient.py#L98-L103) method:

```python
for query in queries:
    params = {
        'api_key': '...',                 # https://serpapi.com/manage-api-key
        'engine': 'youtube',              # search engine
        'device': 'desktop',              # device type
        'search_query': query,            # search query
    }

    search = YoutubeSearch(params)       # where data extraction happens
    results = search.get_dict()          # JSON -> Python dict

```

[Check for 'errors'](https://github.com/serpapi/google-search-results-python#error-management), iterate over video results and extract needed data to the temporary `list`.

```python
if 'error' in results:
    print(results['error'])
    break

```

After that, we need to check for the `'next'` key from `'serpapi_pagination'` and update search parameters data to a data from the next page, or exit the infinite loop if there are no more pages left:

```python
for result in results.get('video_results', []):
    data.append({
        'title': result.get('title'),
        'link': result.get('link'),
        'channel': result.get('channel').get('name')
    })

if 'next' in results.get('serpapi_pagination', {}):
    search.params_dict.update(dict(parse_qsl(urlsplit(results.get('serpapi_pagination', {}).get('next')).query)))
else:
    break

```

## Async Pagination

You can [check the code example in the online IDE](https://replit.com/@serpapi/SerpApi-YouTube-Search-Async-with-Pagination#main.py):

```python
from serpapi import YoutubeSearch
from urllib.parse import (parse_qsl, urlsplit)
from queue import Queue
import json

queries = [
    'object',
    'far-flung',
    'gabby',
    'tiresome',
    'scatter',
    'exclusive',
    'wealth'
]

search_queue = Queue()

for query in queries:
    params = {
        'api_key': '...',                 # https://serpapi.com/manage-api-key
        'engine': 'youtube',              # search engine
        'device': 'desktop',              # device type
        'search_query': query,            # search query
        'async': True,                    # async batch requests
    }

    search = YoutubeSearch(params)       # where data extraction happens
    results = search.get_dict()          # JSON -> Python dict
    
    if 'error' in results:
        print(results['error'])
        break

    print(f"add search to the queue with ID: {results['search_metadata']}")
    search_queue.put(results)

data = []

while not search_queue.empty():
    result = search_queue.get()
    search_id = result['search_metadata']['id']

    print(f'Get search from archive: {search_id}')
    search_archived = search.get_search_archive(search_id) # where all extracted data is stored and accessed
    
    print(f"Search ID: {search_id}, Status: {search_archived['search_metadata']['status']}")

    if re.search(r'Cached|Success', search_archived['search_metadata']['status']):
        for video_result in search_archived.get('video_results', []):
            data.append({
                'title': video_result.get('title'),
                'link': video_result.get('link'),
                'channel': video_result.get('channel').get('name'),
            })
            
        if 'next' in search_archived.get('serpapi_pagination', {}):
            search.params_dict.update(dict(parse_qsl(urlsplit(search_archived['serpapi_pagination']['next']).query)))
            
            new_page_result = search.get_dict() # new results from updated (next) page
            search_queue.put(new_page_result)   # add to queue results from updated (next) page
    else:
        print(f'Requeue search: {search_id}')
        search_queue.put(result)
        
# print(json.dumps(data, indent=2))
print('all searches completed')

```

### Async Pagination Explanation

Import libraries:

```python
from serpapi import YoutubeSearch
from urllib.parse import (parse_qsl, urlsplit) # for pagination
from queue import Queue                        # for async requests 
import json

```

Create a `list` of search queries you want to search:

```python
queries = [
    'object',
    'far-flung',
    'gabby',
    'tiresome',
    'scatter',
    'exclusive',
    'wealth'
]

```

[Create a Queue](https://docs.python.org/3/library/queue.html#module-queue) that will store all requests that have been sent to SerpApi for processing:

```python
search_queue = Queue()

```

Iterate over all queries, create [SerpApi YouTube search parameters](https://serpapi.com/youtube-search-api#api-parameters) with `'async': True` parameter present. Check for errors and [put() search in the queue](https://docs.python.org/3/library/queue.html#queue.Queue.put):

```python
for query in queries:
    params = {
        'api_key': '...',                 # https://serpapi.com/manage-api-key
        'engine': 'youtube',              # search engine
        'device': 'desktop',              # device type
        'search_query': query,            # search query
        'async': True,                    # async batch requests
    }

    search = YoutubeSearch(params)       # where data extraction happens
    results = search.get_dict()          # JSON -> Python dict
    
    if 'error' in results:
        print(results['error'])
        break

    print(f"add search to the queue with ID: {results['search_metadata']}")
    search_queue.put(results)

```

Create a temporary `list` that will be used to store extracted data from the search archive API:

```python
data = []

```

[Iterate through all queue until it's empty()](https://docs.python.org/3/library/queue.html#queue.Queue.empty) and get the data from search archive by accessing search ID:

```python
while not search_queue.empty():
    result = search_queue.get()
    search_id = result['search_metadata']['id']

    print(f'Get search from archive: {search_id}')
    search_archived = search.get_search_archive(search_id) # where all extracted data is stored and accessed
    
    print(f"Search ID: {search_id}, Status: {search_archived['search_metadata']['status']}")

```

Check if the search is either *cached* or *succeeded*, if so, extract the needed data.

After extracting data we need to check if the `'next'` page is present inside `'serpapi_pagination'`. If it's present, we need to assign new page data to a new variable and `put()` it to the `Queue`.

If `'next'` key is not present, we need to exit the pagination and proceed to requeuing result or exit `while` loop depending on the result `'status'`:

```python
if re.search(r'Cached|Success', search_archived['search_metadata']['status']):
        for result in search_archived.get('video_results', []):
            data.append({
                'title': result.get('title'),
                'link': result.get('link'),
                'channel': result.get('channel').get('name')
            })

        if 'next' in search_archived.get('serpapi_pagination', {}):
            search.params_dict.update(dict(parse_qsl(urlsplit(search_archived['serpapi_pagination']['next']).query)))
            
            new_page_result = search.get_dict() # new results from updated (next) page
            search_queue.put(new_page_result)   # add to queue results from updated (next) page
    else:
        print(f'Requeue search: {search_id}')
        search_queue.put(result)
        
print(json.dumps(data, indent=2))
print('all searches completed')

```

## Conclusion

In this comparison, you saw a 430% speed increase when using Async in comparison to Sync requests combined with pagination.

With that said, we strongly recommend using `async` parameter if there's a need to extract data with pagination.

## What comes next

In the next blog post we'll cover:

- how to speed up `async` requests even more.

Join us on [Twitter](https://twitter.com/serp%5Fapi) | [YouTube](https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg%5Fmg)