> ## Content Index > Fetch the complete content index at: https://serpapi.com/blog/llms.txt > Use this file to discover other available public pages before exploring further. # What Is Web Crawling? - URL: https://serpapi.com/blog/what-is-web-crawling/ - Published: 2026-11-09T04:20:26.000Z - Updated: 2026-11-09T04:51:55.000Z - Description: Learn what web crawling is, its use cases, and how crawlers decide which pages to visit. Includes a site crawl use case in Python. - Author: Srujana Maddula The web is enormous, and no one can keep track of every page manually. New pages appear, old ones disappear, and websites constantly change their content. Web crawlers help keep track of all this by automatically discovering and visiting pages across the web or within individual websites. But how exactly do web crawlers do that? How do they decide which pages to visit? And how is crawling different from scraping? I'll answer all these questions in this article, along with a practical Python code example showing how to crawl. ## What is web crawling? A web crawler is a software program that automatically discovers and visits web pages. It starts with one or more seed URLs, fetches those pages, extracts their links, visits the new URLs it finds, and repeats the process within a defined scope. Depending on its purpose, it may also collect information such as page titles, metadata, and links. Search engines like Google use web crawlers to discover pages across the internet before deciding which ones to index. Website owners and developers also use crawlers to audit websites, find broken links, or collect information across multiple pages. The main output of these crawlers is typically a list of discovered URLs, often with metadata like page titles, link relationships, or the content of each page. ## How does web crawling work? Say a crawler starts on a site’s homepage. The homepage links to product pages, blog posts, and category pages. Those pages further link to even more sub pages. So a typical crawling looks like this: 1. **Start with seed URLs.** The crawler receives one or more starting URLs. Depending on the crawler, it may use a sitemap or a previously discovered URL list. 2. **Check robots.txt and fetch the page.** A well-behaved crawler reads the site’s rules first, then requests the permitted page and extracts links. 3. **Queue new URLs.** The crawler filters duplicates and applies rules such as allowed domains, URL patterns, crawl depth, and page limits. 4. **Repeat.** It fetches the next eligible URLs, discovers more links, and continues until it reaches a stopping condition. ## How do crawlers decide which pages to visit? If a crawler follows every link without any limits, it keeps visiting URLs forever. So web crawlers use a few rules to prioritize some pages over others. Here they are: **Inbound links:** If a page has more internal or external links pointing to it, it is prioritized and indexed by search engines. **Site and brand authority:** Search engines prioritize sites with high domain authority and traffic volume for indexing. **Closer to seed URLs:** Most web crawlers use breadth-first as their default strategy. Breadth-first uses a First-In-First-Out (FIFO) queue, which means it crawls the immediate links on the page first. Then it goes to the pages two links away, and so on. **Follows sitemaps:** A sitemap is a file that lists a website’s pages and gives metadata about their structure and importance. Crawlers read these XML sitemaps, identify the important URLs, and crawl them. This way they don’t miss orphan pages, which have no incoming links and stay invisible, or deep pages that take many clicks to reach. **robots.txt:** Most sites publish a robots.txt file with rules about which pages a crawler may or may not visit. A well-behaved crawler follows these rules and skips the pages it isn’t allowed to crawl. ## Web crawling vs web scraping Web crawling is about discovering pages while web scraping extracts information from those pages. | Factor | Web crawling | Web scraping | | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | -------------------------------------------------------- | ------------ | | Purpose | Discover pages and follow links | Extract specific information from pages | | | Input | Seed URLs, sitemaps, or URL patterns | Single URL, or a set of URLs you want to scrape | | | Output | Discovered URLs, metadata, and link relationships | Rendered HTML, or structured content in JSON or markdown | | | Speed | Generally slower, especially when crawling deeper through multiple levels of links | Faster when the target URLs are already known | | | Use cases | Search engine indexing, SEO site audits, etc., | Lead generation, e-commerce data scraping, etc., | | | Web crawling in action: auditing a blog before a migrationYou’re moving a site to a new CMS or domain and need to know every page that exists, so nothing breaks or loses its SEO value. Here's how crawling and mapping helps.Build a URL inventory: Find URLs through internal links and sitemaps so you have a broader picture of the pages on your site.Fetch content: Crawl the important links and pull information like page’s title, headings, and word count.Prepare migration decisions: Organize the results in a spreadsheet with each URL, its crawl depth, title, word count, and a recommended action: keep, redirect, or review.To do all of this with a simple API call, I use the SerpApi Extract API. It has map, crawl, and scrape modes for exactly these tasks. Let's build a script that maps the SerpApi blog, crawls it, and saves the results to audit.csv.Web crawling example in PythonSetupTo run the code below, you need Python, the requests library, and a SerpApi API key saved as the SERPAPI\_API\_KEY environment variable.Install the library: pip install requestsSet your API key as an environment variable:export SERPAPI\_API\_KEY="your\_api\_key"The call\_extract() function sends requests to the Extract API using the parameters passed to it.The normalize() function stops the same page from being counted twice (by removing trailing slashes, fragments), and the in\_section() keeps the audit within the /blog section.import csv | | | | | import hashlib | | | | | import os | | | | | import re | | | | | from collections import Counter, defaultdict | | | | | from urllib.parse import urlsplit, urlunsplit | | | | | import requests | | | | | API\_KEY = os.getenv("SERPAPI\_API\_KEY") | | | | | START\_URL = "https://serpapi.com/blog/" | | | | | SECTION = "/blog" # only audit this path (and its children) | | | | | MAX\_PAGES = 100 # crawl cap | | | | | MAX\_DEPTH = 3 # crawl depth | | | | | MAX\_EXTRA = 50 # extra single-page scrapes for mapped URLs the crawl missed | | | | | THIN\_WORDS = 300 | | | | | CONTENT\_TAG = "article" # extract only the article body | | | | | LISTING\_PATHS = ("/author/", "/page/", "/tag/") # index pages, not content | | | | | def call\_extract(\*\*params): | | | | | params = { | | | | | "engine": "extract", | | | | | "api\_key": API\_KEY, | | | | | "include\_tags": CONTENT\_TAG, | | | | | \*\*params, | | | | | } | | | | | resp = requests.get( | | | | | "https://serpapi.com/search.json", | | | | | params=params, | | | | | timeout=180, | | | | | ) | | | | | resp.raise\_for\_status() | | | | | data = resp.json() | | | | | if data.get("error"): | | | | | raise RuntimeError(data\["error"\]) | | | | | return data | | | | | def normalize(url): | | | | | """Drop fragment and trailing slash; keep the query so distinct pages stay distinct.""" | | | | | p = urlsplit(url) | | | | | return urlunsplit( | | | | | (p.scheme, p.netloc, p.path.rstrip("/") or "/", p.query, "") | | | | | ) | | | | | def in\_section(url): | | | | | path = urlsplit(url).path.rstrip("/") | | | | | return path == SECTION or path.startswith(SECTION + "/")The next code section follows the below steps to run the audit:Map mode asks the API for every URL it finds on serpapi.com, then keeps only the ones under /blog. For each URL, it also saves how it was found, whether from a link on the page (page) or from the sitemap (sitemap).Crawl starts at the blog index and follows links up to MAX\_PAGES and MAX\_DEPTH. Each page is stored with its depth, and it gets a crawl label.If the crawl misses any URLs found during mapping, the script scrapes them individually, up to MAX\_EXTRA. These are usually pages found only in the sitemap or pages the crawl couldn't reach within its limits. Since the crawler never visited these pages, they don't have a crawl depth. Now each page’s markdown gives its word count and headings. An MD5 hash of the content lets us catch duplicates. Inbound links are counted from the internal links on every extracted page.Then it goes through every URL, labels it as keep, redirect, review, or skip, and writes the results to audit.csv.def run\_audit(): | | | | | \# Step 1: map -> inventory with how each URL was found (page / sitemap) | | | | | found\_by = defaultdict(set) | | | | | for item in call\_extract(url=START\_URL, mode="map").get("urls", \[\]): | | | | | url = normalize(item\["url"\]) | | | | | if in\_section(url): | | | | | found\_by\[url\].add(item.get("source", "")) | | | | | \# Step 2: crawl, then scrape mapped URLs the crawl didn't reach (e.g. sitemap-only pages) | | | | | crawled = call\_extract( | | | | | url=START\_URL, | | | | | mode="crawl", | | | | | max\_pages=MAX\_PAGES, | | | | | max\_depth=MAX\_DEPTH, | | | | | ) | | | | | pages = {} # url -> (depth, page data) | | | | | for page in crawled.get("pages", \[\]): | | | | | url = normalize(page\["url"\]) | | | | | if in\_section(url): | | | | | pages\[url\] = (page.get("depth"), page) | | | | | found\_by\[url\].add("crawl") | | | | | missed = \[u for u in found\_by if u not in pages\]\[:MAX\_EXTRA\] | | | | | for url in missed: | | | | | try: | | | | | pages\[url\] = ("", call\_extract(url=url)) | | | | | except (requests.RequestException, RuntimeError) as e: | | | | | print(f"Could not scrape {url}: {e}") | | | | | \# Step 3: analyse | | | | | inbound = Counter() | | | | | info = {} | | | | | for url, (depth, page) in pages.items(): | | | | | md = page.get("markdown") or "" | | | | | info\[url\] = { | | | | | "depth": depth, | | | | | "title": (page.get("page\_information") or {}).get("title", ""), | | | | | "words": len(re.findall(r"\\w+", md)), | | | | | "headings": " | ".join( | | | | h.lstrip("# ").strip() | | | | | for h in re.findall(r"^#{1,6}\\s+.+$", md, re.M) | | | | | ), | | | | | "hash": hashlib.md5(md.encode()).hexdigest() if md else "", | | | | | } | | | | | for link in page.get("inline\_links", \[\]): | | | | | target = normalize(link.get("link", "")) | | | | | if link.get("internal") and target != url: | | | | | inbound\[target\] += 1 | | | | | posts = \[ | | | | | i for u, i in info.items() | | | | | if not any(p in u for p in LISTING\_PATHS) | | | | | \] | | | | | hashes = Counter(i\["hash"\] for i in posts if i\["hash"\]) | | | | | rows = \[\] | | | | | for url in sorted(found\_by): | | | | | i = info.get(url) | | | | | row = { | | | | | "url": url, | | | | | "found\_by": "+".join(sorted(found\_by\[url\])), | | | | | "inbound\_links": inbound\[url\], | | | | | } | | | | | if not i: | | | | | row.update(action="review (could not extract)") | | | | | elif any(p in url for p in LISTING\_PATHS): | | | | | row.update( | | | | | depth=i\["depth"\], | | | | | title=i\["title"\], | | | | | action="skip (listing page)", | | | | | ) | | | | | else: | | | | | if i\["hash"\] and hashes\[i\["hash"\]\] > 1: | | | | | action = "redirect (duplicate)" | | | | | elif i\["words"\] < THIN\_WORDS: | | | | | action = "redirect (thin)" | | | | | else: | | | | | action = "keep" | | | | | row.update( | | | | | depth=i\["depth"\], | | | | | title=i\["title"\], | | | | | word\_count=i\["words"\], | | | | | headings=i\["headings"\], | | | | | action=action, | | | | | ) | | | | | rows.append(row) | | | | | fields = \[ | | | | | "url", | | | | | "depth", | | | | | "title", | | | | | "word\_count", | | | | | "headings", | | | | | "found\_by", | | | | | "inbound\_links", | | | | | "action", | | | | | \] | | | | | with open("audit.csv", "w", newline="", encoding="utf-8") as f: | | | | | writer = csv.DictWriter(f, fieldnames=fields) | | | | | writer.writeheader() | | | | | writer.writerows(rows) | | | | | print( | | | | | f"{len(rows)} URLs inventoried, {len(info)} extracted " | | | | | f"({len(missed)} extra scrapes). Wrote audit.csv" | | | | | ) | | | | | if len(crawled.get("pages", \[\])) >= MAX\_PAGES: | | | | | print("Warning: crawl hit MAX\_PAGES; coverage may be incomplete.") | | | | | if \_\_name\_\_ == "\_\_main\_\_": | | | | | if not API\_KEY: | | | | | raise SystemExit("Set SERPAPI\_API\_KEY first.") | | | | | run\_audit()Popular tools for scrapers and crawlersThere are several tools available for scraping or crawling websites. Some handle both tasks, while others focus on a specific parts, such as discovering URLs, extracting page data, or mapping site URLs. Here's a look at some popular options and what they're best suited for. | Tool category | Best for | Web scraping | | All-in-one crawler and scraper | Mapping, crawling, and scraping any public site through one API | SerpApi Extract API | | | Search engine crawlers | Discovering and indexing pages across the web | Googlebot, Bingbot | | | SEO website crawlers | Auditing internal links, status codes, redirects, and page metadata | SerpApi Extract API, Screaming Frog SEO Spider, Sitebulb | | | Open-source crawling frameworks | Building custom crawling pipelines with full control over scheduling, throttling, and storage | Scrapy, Crawlee | | | AI-ready open-source crawlers | Crawling sites into clean Markdown for LLM, RAG, and AI agent pipelines | Crawl4AI | | | HTML parsing libraries | Extracting data from pages you’ve already fetched | Beautiful Soup, lxml, Cheerio | | | Browser automation tools | Scraping JavaScript-rendered pages, or pages that require browser interactions | Playwright, Selenium, Puppeteer | | | Most of these tools make you choose between building your own scraper/crawler or using a hosted service. Open-source options like Scrapy and Crawl4AI give you control over the crawling process, but you're also responsible for the infrastructure, scheduling, and ongoing maintenance.SerpApi’s Extract API simplifies this. One endpoint covers all three jobs: map to discover a site’s URLs, crawl to extract multiple pages, and scrape to extract full content of a page. The JSON response includes page information, meta tags, structured data like Open Graph and JSON-LD, Markdown content, links, images, and media. You can also request Markdown-only output for LLM workflows. The API also follows each site's robots.txt rules, with map and crawl modes skipping disallowed URLs.Final thoughtsEvery time you search for something on internet, or an AI tool answers a question about a website, a crawler has already visited that page first. You never see it, but it’s the first step in getting pages found and indexed.Not just that. If you want to scrape the content of an entire website, you first need to crawl it to find all the page URLs, and then scrape those pages. I showed one popular use case in this article, but you can try the SerpApi Extract API for yourself with 250 free searches per month. | | | |