What Is Web Crawling?

The web is enormous, and no one can keep track of every page manually. New pages appear, old ones disappear, and websites constantly change their content. Web crawlers help keep track of all this by automatically discovering and visiting pages across the web or within individual websites.

But how exactly do web crawlers do that? How do they decide which pages to visit? And how is crawling different from scraping? I'll answer all these questions in this article, along with a practical Python code example showing how to crawl.

What is web crawling?

A web crawler is a software program that automatically discovers and visits web pages. It starts with one or more seed URLs, fetches those pages, extracts their links, visits the new URLs it finds, and repeats the process within a defined scope. Depending on its purpose, it may also collect information such as page titles, metadata, and links.

Search engines like Google use web crawlers to discover pages across the internet before deciding which ones to index. Website owners and developers also use crawlers to audit websites, find broken links, or collect information across multiple pages.

The main output of these crawlers is typically a list of discovered URLs, often with metadata like page titles, link relationships, or the content of each page.

How does web crawling work?

Say a crawler starts on a site’s homepage. The homepage links to product pages, blog posts, and category pages. Those pages further link to even more sub pages. So a typical crawling looks like this:

1. Start with seed URLs. The crawler receives one or more starting URLs. Depending on the crawler, it may use a sitemap or a previously discovered URL list.
2. Check robots.txt and fetch the page. A well-behaved crawler reads the site’s rules first, then requests the permitted page and extracts links.
3. Queue new URLs. The crawler filters duplicates and applies rules such as allowed domains, URL patterns, crawl depth, and page limits.
4. Repeat. It fetches the next eligible URLs, discovers more links, and continues until it reaches a stopping condition.

How do crawlers decide which pages to visit?

If a crawler follows every link without any limits, it keeps visiting URLs forever. So web crawlers use a few rules to prioritize some pages over others. Here they are:

Inbound links: If a page has more internal or external links pointing to it, it is prioritized and indexed by search engines.

Site and brand authority: Search engines prioritize sites with high domain authority and traffic volume for indexing.

Closer to seed URLs: Most web crawlers use breadth-first as their default strategy. Breadth-first uses a First-In-First-Out (FIFO) queue, which means it crawls the immediate links on the page first. Then it goes to the pages two links away, and so on.

Follows sitemaps: A sitemap is a file that lists a website’s pages and gives metadata about their structure and importance. Crawlers read these XML sitemaps, identify the important URLs, and crawl them. This way they don’t miss orphan pages, which have no incoming links and stay invisible, or deep pages that take many clicks to reach. 

robots.txt: Most sites publish a robots.txt file with rules about which pages a crawler may or may not visit. A well-behaved crawler follows these rules and skips the pages it isn’t allowed to crawl.

Web crawling vs web scraping

Web crawling is about discovering pages while web scraping extracts information from those pages.

Web crawling in action: auditing a blog before a migration

You’re moving a site to a new CMS or domain and need to know every page that exists, so nothing breaks or loses its SEO value. Here's how crawling and mapping helps.

• Build a URL inventory: Find URLs through internal links and sitemaps so you have a broader picture of the pages on your site.
• Fetch content: Crawl the important links and pull information like page’s title, headings, and word count.
• Prepare migration decisions: Organize the results in a spreadsheet with each URL, its crawl depth, title, word count, and a recommended action: keep, redirect, or review.

To do all of this with a simple API call, I use the SerpApi Extract API. It has map, crawl, and scrape modes for exactly these tasks. Let's build a script that maps the SerpApi blog, crawls it, and saves the results to audit.csv.

Web crawling example in Python

Setup

To run the code below, you need Python, the requests library, and a SerpApi API key saved as the SERPAPI_API_KEY environment variable.

Install the library: pip install requests

Set your API key as an environment variable:

export SERPAPI_API_KEY="your_api_key"

The call_extract() function sends requests to the Extract API using the parameters passed to it.

The normalize() function stops the same page from being counted twice (by removing trailing slashes, fragments), and the in_section() keeps the audit within the /blog section.

import csv
import hashlib
import os
import re
from collections import Counter, defaultdict
from urllib.parse import urlsplit, urlunsplit

import requests


API_KEY = os.getenv("SERPAPI_API_KEY")

START_URL = "https://serpapi.com/blog/"
SECTION = "/blog"  # only audit this path (and its children)
MAX_PAGES = 100  # crawl cap
MAX_DEPTH = 3  # crawl depth
MAX_EXTRA = 50  # extra single-page scrapes for mapped URLs the crawl missed
THIN_WORDS = 300
CONTENT_TAG = "article"  # extract only the article body
LISTING_PATHS = ("/author/", "/page/", "/tag/")  # index pages, not content


def call_extract(**params):
    params = {
        "engine": "extract",
        "api_key": API_KEY,
        "include_tags": CONTENT_TAG,
        **params,
    }
    resp = requests.get(
        "https://serpapi.com/search.json",
        params=params,
        timeout=180,
    )
    resp.raise_for_status()
    data = resp.json()

    if data.get("error"):
        raise RuntimeError(data["error"])

    return data


def normalize(url):
    """Drop fragment and trailing slash; keep the query so distinct pages stay distinct."""
    p = urlsplit(url)
    return urlunsplit(
        (p.scheme, p.netloc, p.path.rstrip("/") or "/", p.query, "")
    )


def in_section(url):
    path = urlsplit(url).path.rstrip("/")
    return path == SECTION or path.startswith(SECTION + "/")

The next code section follows the below steps to run the audit:

• Map mode asks the API for every URL it finds on serpapi.com, then keeps only the ones under /blog. For each URL, it also saves how it was found, whether from a link on the page (page) or from the sitemap (sitemap).
• Crawl starts at the blog index and follows links up to MAX_PAGES and MAX_DEPTH. Each page is stored with its depth, and it gets a crawl label.
• If the crawl misses any URLs found during mapping, the script scrapes them individually, up to MAX_EXTRA. These are usually pages found only in the sitemap or pages the crawl couldn't reach within its limits. Since the crawler never visited these pages, they don't have a crawl depth. 
• Now each page’s markdown gives its word count and headings. An MD5 hash of the content lets us catch duplicates. Inbound links are counted from the internal links on every extracted page.
• Then it goes through every URL, labels it as keep, redirect, review, or skip, and writes the results to audit.csv.

def run_audit():
    # Step 1: map -> inventory with how each URL was found (page / sitemap)
    found_by = defaultdict(set)

    for item in call_extract(url=START_URL, mode="map").get("urls", []):
        url = normalize(item["url"])
        if in_section(url):
            found_by[url].add(item.get("source", ""))

    # Step 2: crawl, then scrape mapped URLs the crawl didn't reach (e.g. sitemap-only pages)
    crawled = call_extract(
        url=START_URL,
        mode="crawl",
        max_pages=MAX_PAGES,
        max_depth=MAX_DEPTH,
    )
    pages = {}  # url -> (depth, page data)

    for page in crawled.get("pages", []):
        url = normalize(page["url"])
        if in_section(url):
            pages[url] = (page.get("depth"), page)
            found_by[url].add("crawl")

    missed = [u for u in found_by if u not in pages][:MAX_EXTRA]

    for url in missed:
        try:
            pages[url] = ("", call_extract(url=url))
        except (requests.RequestException, RuntimeError) as e:
            print(f"Could not scrape {url}: {e}")

    # Step 3: analyse
    inbound = Counter()
    info = {}

    for url, (depth, page) in pages.items():
        md = page.get("markdown") or ""
        info[url] = {
            "depth": depth,
            "title": (page.get("page_information") or {}).get("title", ""),
            "words": len(re.findall(r"\w+", md)),
            "headings": " | ".join(
                h.lstrip("# ").strip()
                for h in re.findall(r"^#{1,6}\s+.+$", md, re.M)
            ),
            "hash": hashlib.md5(md.encode()).hexdigest() if md else "",
        }

        for link in page.get("inline_links", []):
            target = normalize(link.get("link", ""))
            if link.get("internal") and target != url:
                inbound[target] += 1

    posts = [
        i for u, i in info.items()
        if not any(p in u for p in LISTING_PATHS)
    ]
    hashes = Counter(i["hash"] for i in posts if i["hash"])

    rows = []

    for url in sorted(found_by):
        i = info.get(url)
        row = {
            "url": url,
            "found_by": "+".join(sorted(found_by[url])),
            "inbound_links": inbound[url],
        }

        if not i:
            row.update(action="review (could not extract)")
        elif any(p in url for p in LISTING_PATHS):
            row.update(
                depth=i["depth"],
                title=i["title"],
                action="skip (listing page)",
            )
        else:
            if i["hash"] and hashes[i["hash"]] > 1:
                action = "redirect (duplicate)"
            elif i["words"] < THIN_WORDS:
                action = "redirect (thin)"
            else:
                action = "keep"

            row.update(
                depth=i["depth"],
                title=i["title"],
                word_count=i["words"],
                headings=i["headings"],
                action=action,
            )

        rows.append(row)

    fields = [
        "url",
        "depth",
        "title",
        "word_count",
        "headings",
        "found_by",
        "inbound_links",
        "action",
    ]

    with open("audit.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fields)
        writer.writeheader()
        writer.writerows(rows)

    print(
        f"{len(rows)} URLs inventoried, {len(info)} extracted "
        f"({len(missed)} extra scrapes). Wrote audit.csv"
    )

    if len(crawled.get("pages", [])) >= MAX_PAGES:
        print("Warning: crawl hit MAX_PAGES; coverage may be incomplete.")


if __name__ == "__main__":
    if not API_KEY:
        raise SystemExit("Set SERPAPI_API_KEY first.")

    run_audit()

Popular tools for scrapers and crawlers

There are several tools available for scraping or crawling websites. Some handle both tasks, while others focus on a specific parts, such as discovering URLs, extracting page data, or mapping site URLs. Here's a look at some popular options and what they're best suited for.

Most of these tools make you choose between building your own scraper/crawler or using a hosted service. Open-source options like Scrapy and Crawl4AI give you control over the crawling process, but you're also responsible for the infrastructure, scheduling, and ongoing maintenance.

SerpApi’s Extract API simplifies this. One endpoint covers all three jobs: map to discover a site’s URLs, crawl to extract multiple pages, and scrape to extract full content of a page. The JSON response includes page information, meta tags, structured data like Open Graph and JSON-LD, Markdown content, links, images, and media. You can also request Markdown-only output for LLM workflows. The API also follows each site's robots.txt rules, with map and crawl modes skipping disallowed URLs.

Final thoughts

Every time you search for something on internet, or an AI tool answers a question about a website, a crawler has already visited that page first. You never see it, but it’s the first step in getting pages found and indexed.

Not just that. If you want to scrape the content of an entire website, you first need to crawl it to find all the page URLs, and then scrape those pages. I showed one popular use case in this article, but you can try the SerpApi Extract API for yourself with 250 free searches per month.