Sign In Register
0 searches
+ 0 extra credits

Extract Crawl Mode API

Crawl mode follows links on the same website, level by level, and returns every page it visits in pages, with the same fields as a scrape result plus its url and depth. It stops at max_pages or max_depth.

API Parameters

Search Query

url

Required

Parameter defines the URL of the page to extract. It must be an absolute http or https URL. Search pages that have a dedicated SerpApi engine, such as Google or Bing, are redirected to that engine.

Mode

mode

Optional

Parameter defines the extraction mode. It can be set to:
scrape (default) - Extract a single page.
map - List the website's URLs from the page links and sitemaps.
crawl - Follow links on the same website and extract every page visited.

Crawl mode consumes one search credit per successfully extracted page.

Crawl Settings

max_pages

Optional

Used with mode=crawl. Parameter defines the maximum number of pages to extract, including the starting page. Default: 10. Maximum: 100. Each successfully extracted page consumes one search credit.

max_depth

Optional

Used with mode=crawl. Parameter defines how many links deep the crawl follows from the starting page. 0 extracts only the starting page. Default: 2. Maximum: 3.

include_subdomains

Optional

Used with mode=map and mode=crawl. Parameter defines whether to include other subdomains of the same website, e.g. blog.example.com when starting from www.example.com. Default: false (same host only).

Crawl Credit Usage

Crawl mode consumes one search credit per successfully extracted page, including the starting page, so max_pages=10 can cost up to 10 credits. Failed pages and cached responses are not charged. search_metadata.credits_consumed shows the cost.

API Examples

Crawl an origin mode: crawl

Crawl an origin mode: crawl

JSON Example

{
  ...
  "search_metadata": {
    "credits_consumed": 3
  },
  "search_information": {
    "organic_results_state": "Crawl complete"
  },
  "pages": [
    {
      "position": 1,
      "url": "https://books.toscrape.com/",
      "depth": 0,
      "serpapi_extract_link": "https://serpapi.com/search.json?device=desktop&engine=extract&max_depth=1&max_pages=3&url=https%3A%2F%2Fbooks.toscrape.com%2F",
      "page_information": { "title": "All products | Books to Scrape - Sandbox" },
      "headings": [{ "position": 1, "level": 1, "title": "All products" }]
    },
    {
      "position": 2,
      "url": "https://books.toscrape.com/index.html",
      "depth": 1,
      "serpapi_extract_link": "https://serpapi.com/search.json?device=desktop&engine=extract&max_depth=1&max_pages=3&url=https%3A%2F%2Fbooks.toscrape.com%2Findex.html",
      "page_information": { "title": "All products | Books to Scrape - Sandbox" }
    },
    {
      "position": 3,
      "url": "https://books.toscrape.com/catalogue/category/books_1/index.html",
      "depth": 1,
      "serpapi_extract_link": "https://serpapi.com/search.json?device=desktop&engine=extract&max_depth=1&max_pages=3&url=https%3A%2F%2Fbooks.toscrape.com%2Fcatalogue%2Fcategory%2Fbooks_1%2Findex.html",
      "page_information": { "title": "Books | Books to Scrape - Sandbox" }
    }
  ]
}