How to Scrape DuckDuckGo Search Results

DuckDuckGo returns different results than Google and Bing. It's now one of the top search engine people use in the internet.

When you need results with prioritized user privacy, scraping DuckDuckGo results can be useful. Besides that, you can scrape DuckDuckGo results to do competitor analysis, improve your SEO, and aggregate content.

In my previous blog post, we explored how we can easily scrape Google and Bing results with APIs from SerpApi. It would be easy with DuckDuckGo too.

Here is a video tutorial on how to scrape DuckDuckGo results in Python:

Preparation

You can play around with our playground feature first to see all the response we return on our API.

Get your API Key
Once you are familiar with all the results, you can register a free acount at serpapi.com to get your API key. You can use this API Key to access all of our APIs, including the DuckDuckGo search API.

cURL Implementation

Here is the basic implementation in cURL:

curl --get https://serpapi.com/search \
 -d engine="duckduckgo" \
 -d q="Apple" \
 -d kl="us-en" \
 -d api_key="YOUR_API_KEY"

Scrape your first DuckDuckGo Search Result with SerpApi

Let's start calling the API in Python.

• Create a new main.py file
• Install requests with:

pip install requests

Here is what the basic setup looks like:

import requests
SERPAPI_API_KEY = "YOUR_REAL_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, #replace with your real API Key
    # soon
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

With these few lines of code, we can access all of the search engines available at SerpApi, including the DuckDuckGo Search API

import requests
SERPAPI_API_KEY = "YOUR_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, 
    "engine" : "duckduckgo",
    "q": "Why is DuckDuckGo better?",
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

To make the response easier to see, let's add indentation to the output.

import json

# ...
# ...
# all previous code

print(json.dumps(response, indent=4))

You can always refer to the DuckDuckGo Search API documentation for details.

Running this Python file should show you the search results:

Export to a CSV file

You can store DuckDuckGo Search Results JSON data in databases or export it to a CSV file.

import csv

...
results = response["organic_results"]

header = ['position', 'title', 'link', 'favicon', 'snippet']

with open('duckduckgo.csv', 'w', encoding='UTF8', newline='') as f:
    writer = csv.writer(f)

    writer.writerow(header)

    for item in results:
        print(item)
        writer.writerow([item.get('position'), item.get('title'), item.get('link'), item.get('favicon'), item.get('snippet')])

This example is using Python, but you can also use all your favorite programming languages like Ruby, NodeJS, Java, PHP, and more.

Scrape search assist (AI overview) answer

SEO is not just about organic results anymore, there are many other points on the search results that useful to track in this AI era.

For example, DuckDuckGo offers "Search Assist" answer for visitors to find answers quickly:

In order to get this result, you just need to include "search_assist": True in the parameter. For example:

params = {
    "api_key": SERPAPI_API_KEY,
    "engine" : "duckduckgo",
    "q": "Why is DuckDuckGo better?",
    "search_assist": True
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()

print(json.dumps(response, indent=2))

You should see the search_assist answer before the organic_results

Check more API you can use for Generative Engine Optimization.