> ## Content Index
> Fetch the complete content index at: https://serpapi.com/blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# Scraping Business Listings from Duck Duck Go Maps with NodeJS
- URL: https://serpapi.com/blog/scraping-duck-duck-go-maps-with-javascript/
- Published: 2023-10-20T22:40:36.000Z
- Updated: 2023-11-02T18:20:38.000Z
- Description: Introducing the DuckDuckGo Maps API
- Author: Ryan Schafer
- Tags: node, NodeJS, JavaScript, Google Maps, DuckDuckGo, business listing

## Intro

SerpApi released a new [DuckDuckGo Maps API](https://serpapi.com/duckduckgo-maps-api) in August. You can use it to scrape location details from DuckDuckGo Maps. 

In this tutorial, we will discuss some of the advantages the DuckDuckGo Maps API offers, and create an implementation in JavaScript/Node.js based on a common use case - we will build a script to find all of the Coffee Shops in Austin, Texas.

If you do not need an explanation and just want to download and use the application, you can access the [full code in the GitHub repository](https://github.com/schaferyan/Duck-Maps).

## Advantages of SerpApi

SerpApi is a web-scraping API that streamlines the process of scraping data from search engines. This can also be done manually, but SerpApi provides several distinct advantages. 

Besides providing real-time data in a convenient JSON format, SerpApi covers proxies, captcha solvers, and other necessary techniques for avoiding searches being blocked. You can [click here](https://dev.to/mikhailzub/how-to-scrape-google-search-organic-results-with-node-js-4id9) for a comparison of the process of scraping Google Search Results manually vs using SerpApi, or [here](https://serpapi.com/blog/how-to-reduce-chance-of-being-blocked-while-web/) for an overview of the techniques you can use to prevent getting blocked.

## Advantages of the DuckDuckGo Maps API

For SerpApi users who are already familiar with SerpApi's Google Maps API, you may be wondering what the benefits of the DuckDuckGo Maps API are.

The primary advantage the Duck Duck Go Maps API has over the Google Maps API is its unique `bbox_parameter`, which allows you to set specific boundaries for your search. 

Let's take a look at a common use case. You want to scrape business listings in a particular city, and you want to make sure you get them all. You'd also like to avoid any duplicates, so as not to waste your searches.

With Google Maps, the only option is to increase the zoom parameter to its maximum value, estimate the radius of the search, and keep changing the GPS coordinates to try to cover the entire area.

The trouble with this is that Google doesn't provide a consistent search radius. If there are few results available near the GPS coordinates, you're going to get some results further away. This makes it impossible to cover an area with the highest precision, without having to filter out duplicates.

With DuckDuckGo Maps on the other hand, we have a `bbox_parameter` and a `strict_bbox` parameter. When `strict_bbox` is set to 1, only results within the defined area will be returned.

## Prerequisites

- You will need to be somewhat familiar with JavaScript. You will have an easier time if you are familiar with ES6 syntax and features, as well as Node and Npm.
- You need to have the latest versions of Node and Npm installed on your system. If you don’t already have Node and Npm installed you can visit the following link for help with this process: <https://docs.npmjs.com/downloading-and-installing-node-js-and-npm>
- You need to have an IDE installed on your system that supports JavaScript and Node.js. I recommend VSCode or Sublime, but any IDE that supports Node.js will do.
- You will also need to sign up for a free SerpApi account at <https://serpapi.com/users/sign%5Fup>.

## Preparation

First, you will need to create a **package.json** file. Open a terminal window, create a directory for the project, and CD into the directory.

```bash
mkdir wallpaper-changer
cd wallpaper-changer

```

Create your **package.json** file:

```bash
npm init

```

Npm will walk you through the process of creating a **package.json** file.  
After this we need to make a few changes to the **package.json** file.  
We will be using the ES Module system rather than CommonJs, so you need to add the line `type="module"` to your package.json.

```json
{
  "name": "wallpaper-changer",
  "version": "1.0.0",
  "description": "Change your wallpaper to a randomized image from the web",
  "type": "module",
  "main": "index.js",

```

We will also add a start script for convenience:

```json
  "scripts": {
    "start": "node index.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },

```

Next, we need to install our dependencies. For this project, we will use the [SerpApi for JavaScript/TypeScript Node module](https://www.npmjs.com/package/serpapi), and the [dotenv](https://www.npmjs.com/package/dotenv) module. If you're not familiar with [dotenv](https://www.npmjs.com/package/dotenv), it is a module for handling environment variables (like file paths and API keys). 

```bash
npm install serpapi dotenv

```

If you haven’t already signed up for a free Serpapi account go ahead and do that now by visiting <https://serpapi.com/users/sign%5Fup> and completing the signup process.

Once you have signed up, verified your email, and selected a plan, navigate to <https://serpapi.com/manage-api-key> . Click the Clipboard icon to copy your API key.

Then create a new file in your project directory called **.env** and add the following line:

```
SERPAPI_KEY = “PASTE_YOUR_API_KEY_HERE”

```

## Scraping DuckDuckGo Maps Search Results with the SerpApi DuckDuckGo Maps API

We're ready to write the code. Let's start with a simple call to the DuckDuck Go Maps API.

Create a file in your project directory called `index.js` and add the following code:

```javascript
import { config, getJson} from "serpapi";
import * as dotenv from "dotenv";

dotenv.config();
config.api_key = process.env.SERPAPI_KEY;

const params = {
  engine: "duckduckgo_maps",
  q: "Coffee",
bbox:"30.341552964181687,-97.87405344947078,30.16321730812698,-97.50702877159034",
  strict_bbox: "1"
}

function getResults(){
  getJson( params, (results) => {
  console.log(results);
  });
}

getResults();

```

Run `npm start` in your terminal. A list of coffee shops and their details should be printed.

Now that we know the call to SerpApi is working, let's remove the hardcoded `q` and `bbox` parameters and add them as parameters in our `getResults()` function:

```javascript
import { config, getJson} from "serpapi";
import * as dotenv from "dotenv";

dotenv.config();
config.api_key = process.env.SERPAPI_KEY;

const params = {
  engine: "duckduckgo_maps",
  strict_bbox: "1"
}

function getResults(q, bbox){
  params.q = q;
  params.bbox = bbox;
  getJson( params, (results) => {
  console.log(results);
  });
}
```

## Searching in a Grid Pattern

The limitation of DuckDuckGo Maps is that it never returns more than 20 results per search and it doesn't have a pagination feature. So for it to be effective in getting all possible results, you need to zoom in close enough that you can be sure there aren't more than 20 results available. You then need to iterate over the area you want to search in a grid pattern.

### Determining the GPS Coordinates

To get all of the coffee shops in Austin, Texas, we first need to define the GPS coordinates for Austin. We can get these by opening a view of Austin in Google Maps, and right-clicking where we want the corners to be.

In a real-life application, we might be more particular about defining the area we want to search, but for this example, we will use the area shown below:

![](https://storage.ghost.io/c/a5/00/a5004977-0dd2-4bcd-9292-dd0e05d4c59e/content/images/2023/10/Austin-with-bbox.png)

### Calculating the Grid Size

Now we need to determine the optimal grid size. We open DuckDuckGo in the browser and search for "Austin, TX". 

We want to find all of the coffee shops in Austin, so we search for "Coffee".

We then navigate to the area where we expect to have the most dense population of results. Let's assume downtown Austin has the most Coffee Shops. We navigate to downtown Austin in the map window, zoom in, and click the "Search this area" button at the top of the map.

![](https://storage.ghost.io/c/a5/00/a5004977-0dd2-4bcd-9292-dd0e05d4c59e/content/images/2023/10/Screenshot-2023-10-20-at-3.23.46-PM.png)

There should be 20 results the first time we do this, if not, we need to zoom out.

![](https://storage.ghost.io/c/a5/00/a5004977-0dd2-4bcd-9292-dd0e05d4c59e/content/images/2023/10/Screenshot-2023-10-18-at-1.08.28-PM-1.png)

Then we zoom in little by little, clicking the "Search this area" button until only 19 results are returned. We move the map window around a little bit and click the button again to make sure nowhere returns 20 results.

Now that we've determined the appropriate zoom distance for getting the maximum number of results, we need to extract the `bbox` value from the URL in the address bar to determine the size of our latitude and longitude offsets.

![](https://storage.ghost.io/c/a5/00/a5004977-0dd2-4bcd-9292-dd0e05d4c59e/content/images/2023/10/Screenshot-2023-10-18-at-1.21.50-PM.png)

The `bbox` parameter includes two sets of GPS coordinates. The first latitude-longitude pair represents the top left corner of a box, and the second pair represents the bottom right corner.

If we want to move from top-left to bottom right in a grid pattern, we need to subtract the second pair from the first to get the offsets we need. So if our `bbox` parameter is `bbox=30.275933298683846,-97.75356226769851,30.26222921173649,-97.73221544030126` we need to do the following calculations:

```
30.26222921173649 - 30.275933298683846 = 0.01370408694
-97.73221544030126 - -97.75356226769851 = -0.02134682739
```

### Looping Over the Grid

For simplicity, let's define some constants for the area we want to search and the latitude and longitude offsets:

```javascript
const area = [30.43874, -97.873607,30.195627,-97.624932]; 
const lat_offset = -0.01370408694;
const lon_offset = 0.02134682739;
```

Now add the following function:

```javascript
function findAll(q, area, lat_offset, lon_offset){
 const coords = [area[0], area[1], area[0] + lat_offset, area[1] + lon_offset];

 /loop while the top part of our window is above and to the left of the bottom and right borders of the area we want to search/
 
 while(coords[0] > area[2]){
  while(coords[1] < area[3]){
    const coords_string = coords.toString();
    console.log(coords_string);
    getResults(q, coords_string);
    coords[1] += lon_offset;
    coords[3] += lon_offset;
  }
  coords[1] = area[1];
  coords[3] = area[3];
  coords[0] += lat_offset;
  coords[2] += lat_offset;
 }

}
```

In the code above, we first define a `coords` array with the GPS coordinates for the first grid section we want to search in.

We then create a nested `while` loop to move through all of the grid squares from top left to bottom right.

Before adding a call to the `findAll()` function and running it from the terminal, we need to make a few more changes.

Right now, the `getResults()` function will print the full JSON response for each search. Let's modify it to just print the number of results for each search, and the names of all of the locations returned:

```javascript
function getResults(q, bbox){
  params.q = q;
  params.bbox = bbox;
  getJson( params, (results) => {
    const local_results = results.local_results;
    if(local_results){
      console.log(local_results.length - 1 + " results");
      for (const result of local_results){
        console.log(result.title);
      }
    }
  });
}
```

Another consideration is that we've defined very small grid squares for our search, meaning it will use a lot of search credits to complete. Before running the code with the offsets we calculated, let's test it with some larger offsets to reduce the number of search credits used:

```javascript
findAll("coffee", area, 0.5, 0.5);
```

After adding the call to `findAll`, type `npm start` into your terminal.

If you did everything correctly, you should see a list of coffee shops in Austin printed out.

 Now to get the maximum number of coffee shops, let's use the constants we defined for `lat_offset` and `lon_offset`:

```javascript
findAll("coffee", area, lat_offset, lon_offset);
```

If you run this, you should see a long list of results printed in the terminal, and you shouldn't see any duplicates.

## Conclusion

We have seen how to scrape local results from DuckDuckGo Maps using the DuckDuckGoMaps API, and how to use the `bbox` parameter to iterate over a grid and get as many results as possible. You now have a template you can modify in any way you need.

I hope you found the tutorial informative and easy to follow. If you have any questions, feel free to contact me at ryan@serpapi.com.

## Links:

[Full code on Github](https://github.com/schaferyan/serpapi-google-search-rankings-nodejs)  
[SerpApi DuckDuckGo Maps API Documentation](https://serpapi.com/duckduckgo-maps-api)  
[](https://serpapi.com/playground)[SerpApi Playground](https://serpapi.com/playground?engine=duckduckgo%5Fmaps&q=Coffee&bbox=30.341552964181687%2C-97.87405344947078%2C30.16321730812698%2C-97.50702877159034&strict%5Fbbox=1)