> ## Content Index > Fetch the complete content index at: https://serpapi.com/blog/llms.txt > Use this file to discover other available public pages before exploring further. # Vessel 0.3: Ruby finally gets its Scrapy crawling framework - URL: https://serpapi.com/blog/vessel-ruby-crawling-framework/ - Published: 2026-09-17T08:34:46.000Z - Updated: 2026-09-17T08:41:51.000Z - Description: The crawling and scraping Vessel project is back with pluggable drivers, the fields API, middleware pipeline, proxy rotation, cookies, retries, and callbacks. See what's new and how to use Vessel to crawl any website using a real Chrome browser or plain HTTP requests. - Author: Josef Strzibny - Tags: Web Scraping, crawling Python developers have Scrapy, one of the [best tool for web scraping](https://serpapi.com/blog/best-web-scraping-tools/) out there. And us Rubyists? We have to piece individual pieces together. Nokogiri, Ferrum, Mechanize, Faraday... but no framework that ties fetching, parsing, concurrency, deduplication, retries, and data pipelines together. Vessel was supposed to be that framework until it went a little quiet. Now it is back. [Vessel 0.3](https://github.com/rubycdp/vessel) is the first release in four years, and it is not a maintenance bump. The internals were rewritten and the feature list finally reads like a real crawling framework. We got pluggable drivers (real Chrome or plain HTTP), a fields API, a middleware pipeline, proxy rotation, cookies, retries, callbacks, and a CLI that generates whole projects. ## What is a crawling framework? A crawling framework bundles scraping and web crawling concerns into one cohesive experience. Tthe framework schedules requests, fetches pages concurrently, deduplicates URLs, retries failures, and pushes whatever you extract through a processing pipeline. In other words, an HTTP client fetches a page, a crawler follows links, but a crawling framework runs the whole loop from scheduling crawls, fetching pages, parsing sources, and storing data. Vessel is a high-level web crawling framework for Ruby. You only really subclass `Vessel::Cargo`, declare a domain and start URLs, and write handler methods that extract data and yield new requests. Scheduling, concurrency, visiting every URL only once, retrying network errors, and pushing extracted items through a processing pipeline is all done for you by Vessel. This is a stark contrast to simple libraries that handle only some of these concerns and makes the user to stitch everything together. ### Scraping a page vs. crawling a site Scraping one page is easy in Ruby. Fetch the HTML, parse it with Nokogiri, done. I covered all of that in my [complete guide to web scraping with Ruby](https://serpapi.com/blog/ruby-web-scraping). Crawling a site is a different problem. You are not extracting data from a page, you are extracting data from a *graph* of pages, and suddenly you need: - A queue of URLs to visit - Deduplication so you do not visit the same URL twice - Concurrency so the crawl finishes this week - Retries for pages that time out - Rate limiting to respect a given site limits You can hand-roll all of this with a `Queue`, a `Set`, and a thread pool. Many of you likely have. But this is exactly the kind of plumbing a framework like Scrapy offered for years. ### Where Spidr and Anemone fall short Ruby does have crawling gems. Don't worry, I know. Spidr and Anemone will happily walk every link on a site and hand you each page. They are fine for link checking or building a URL inventory. I used Spidr before for this kind of tasks. But they are *crawlers*, not *crawling frameworks*. They give you pages but lacks extraction structure, data pipelines, JavaScript rendering, proxies, and per-request state. Anemone has not seen a release in years, and neither runs JavaScript, so any site that renders content client-side is out of reach. ### How Vessel compares to Scrapy Vessel is not a 1:1 copy of Scrapy but there are similarities. We could say the core mental model is now genuinely the same: | Concept | Scrapy | Vessel 0.3 | | ----------------- | ------------------------------------- | ------------------------------------ | | Spider class | scrapy.Spider | Vessel::Cargo | | Entry points | start\_urls | start\_urls (with per-URL handlers) | | Callback loop | yield Request(...) / yield item | yield request(...) / yield hash | | Item pipelines | ITEM\_PIPELINES | middleware "Sanitize", "Save" | | Items and loaders | Item \+ processors | fields API + FieldType normalization | | Dedup filter | on by default | once: true by default | | Retries | retry middleware | network\_error\_attempts | | Project and CLI | startproject, genspider, crawl, parse | vessel new, generate, start, parse | | Stats | stats collector | stats hash + callbacks | And in two places Vessel ships more than Scrapy does out of the box: - **A real browser by default.** Scrapy is HTTP-first, and JavaScript rendering means bolting on scrapy-playwright or Splash. Vessel's default driver is actual Chrome via Ferrum, with a plain-HTTP Mechanize driver when you do not need it. Vessel is a browser-first. - **Built-in proxy rotation.** Round-robin and shuffled rotation ship with the gem while in Scrapy that is a third-party middleware. However, Vessel isn't a Scrapy 1:1 replacement even if they share the *programming model*. Scrapy's scale and ecosystem is still better: - There is no downloader middleware layer. Vessel's middleware is an item pipeline only. There is no global hook to rewrite requests and responses, which is Scrapy's killer extension point. - There are no feed exports. Nothing built-in dumps items to CSV, JSON, or S3 — you write a `Save` middleware yourself. - There is no HTTP cache, no AutoThrottle, and no per-domain concurrency controls. Vessel's `delay` only applies when the crawler runs single-threaded. - There is no robots.txt handling. - The concurrency model is a thread pool, one Chrome page per core by default. Scrapy's async engine comfortably crawls millions of pages while Vessel is built for hundreds to thousands. - Scrapy has fifteen years of plugins, docs, scrapyd, and scrapy-redis behind it. Vessel tracks its first release to 2021. For most Ruby scraping jobs, the model is what was missing, and Vessel now has it. But if you are running a big Scrapy cluster, you might still need to keep it. ## A first crawler in 30 lines Here is the canonical example, crawling *quotes.toscrape.com* with pagination: ```ruby require "json" require "vessel" class QuotesToScrapeCom < Vessel::Cargo domain "quotes.toscrape.com" start_urls "https://quotes.toscrape.com/tag/humor/" def parse css("div.quote").each do |quote| yield({ author: quote.at_xpath("span/small").text, text: quote.at_css("span.text").text }) end next_page = at_xpath("//li[@class='next']/a[@href]") return unless next_page yield request(url: absolute_url(next_page[:href]), handler: :parse) end end quotes = [] QuotesToScrapeCom.run { |q| quotes << q } puts JSON.generate(quotes) ``` Save it as `quotes.rb`, run `bundle exec ruby quotes.rb > quotes.json`, and you have every humor quote across all pages. Vessel visits the start URL and calls the handler (`parse` by default). Inside a handler you query the page directly with `css`, `at_css`, `xpath`, and `at_xpath`. Yielding a hash emits an item while yielding a `request` schedules another page, handled concurrently by a thread pool sized to your cores. The same URL is never visited twice, and pagination is just a handler yielding a request back to itself. 💡 TIP: If you find `Vessel::Cargo` too nautical, `Vessel::Crawler` is an alias. ## What's new in 0.3 ### Pluggable and custom drivers Pluggable drivers give you Chrome when you need it, plain HTTP when you don't. Pages are now fetched by a pluggable driver: ```ruby class MyScraper < Vessel::Cargo driver :ferrum, headless: true, timeout: 30 # or driver :mechanize end ``` `:ferrum` is the default: a real Chrome, JavaScript and all, with sensible crawling defaults (certificate errors ignored, JS errors swallowed, generous timeouts). `:mechanize` is plain HTTP so no browser process, no JavaScript, much faster and lighter. Custom drivers are supported too. Subclass `Vessel::Driver`, implement `start`, `stop`, and `create_page`, and register it. One Ferrum-only nicety: `blacklist` and `whitelist` patterns control which resources Chrome loads, so you can skip images, fonts, and trackers: ```ruby class MyScraper < Vessel::Cargo blacklist [/\.png$/, /googletagmanager/] end ``` This can nicely speed up the overall crawling. ### The fields API with normalization Instead of assembling hashes by hand, handlers can declare fields that gets automatically normalized: ```ruby def parse field :author, value: at_xpath("span/small").text field :text, value: at_css("span.text").text field :html, value: nil, service: true do raw end yield fields end ``` `service: true` keeps a field out of the resulting item but available to the middleware, handy for carrying the raw HTML along for debugging. Fields can be renamed, and `FieldType` normalizes fields by name across all crawlers in one place: ```ruby Vessel::Cargo::FieldType.add(:price) { |value| value.to_s.gsub(/[^\d.]/, "").to_f } ``` Every `field :price` in every crawler now comes out as a Float. If you have ever maintained five scrapers that each clean prices slightly differently, you know why this exists. It is a lighter take on Scrapy's item loaders and processors. ### Middleware as a pipeline Everything a handler yields that is not a request goes through the middleware pipeline, which runs in its own thread pool. A middleware is a class with a `call(hash, fields)` method: ```ruby class Sanitize < Vessel::Middleware def call(hash, fields) hash.transform_values { |v| v.is_a?(String) ? v.strip : v } end end class Save < Vessel::Middleware def call(hash, fields) raise Vessel::Middleware::InvalidItemError if hash[:text].to_s.empty? DB[:quotes].insert(hash) hash end end class MyScraper < Vessel::Cargo middleware "Sanitize", "Save" end ``` Each middleware receives the hash from the previous one plus the original fields object. Raising `InvalidItemError` silently drops an item. This is Scrapy's item pipeline, down to the drop-item semantics. Validation, cleaning, and persistence live here instead of being tangled into your parse handlers. For quick scripts, a block passed to `.run` replaces the whole pipeline, as in the first example. ### Proxy rotation and cookies Proxy rotation is built in. Subclass `Vessel::RoundRobinProxy` or `Vessel::ShuffledProxy`, define a `PROXIES` constant, and the driver takes the next proxy for every page it creates: ```ruby class MyProxy < Vessel::ShuffledProxy PROXIES = [ { host: "127.0.0.1", port: 8080, user: "user1", password: "password1" }, { host: "127.0.0.1", port: 8081, user: "user2", password: "password2" } ].freeze end class MyScraper < Vessel::Cargo proxy MyProxy end ``` Cookies got a proper API as well: `cookie` and `cookies` set them up front, cookies received from responses are kept for subsequent requests by default, and `allow_cookies false` turns that off. Headers, cookies, and delays can also be overridden per request, and requests carry an arbitrary `data` hash over to the response: ```ruby def parse yield request(url: "/page/2/", handler: :parse_page, data: { category: "humor" }) end def parse_page puts response.data[:category] # => "humor" end ``` That `data` hash is Scrapy's `meta`, and it solves the classic crawling problem of carrying context (which category page did this product come from?) across requests. ### Retries, dedup, and callbacks A request that fails with a network error (timeout, socket error, bad status) is retried, five times by default, with the browser restarted between attempts: ```ruby class MyScraper < Vessel::Cargo network_error_attempts 5 end ``` When the attempts run out, the error lands in the `on_error(request, error)` callback, one of a set of new lifecycle hooks (`before_start`, `before`, `after_change`, `info`, `after`, `before_stop`). The `info` callback fires every few seconds with a stats hash (requests enqueued, items processed, items rejected), which is exactly what you want to log in a long crawl. Deduplication is now on by default: the same URL is not visited twice unless you pass `once: false`. And 0.3 fixed a subtle race where threads competing for the same URL could visit it in parallel. ## The new Vessel CLI Vessel comes with a CLI to list crawlers, inspect their settings, and run them. To define a crawler, a single file is usually fine for one. For a collection of them, Vessel now generates a small project for you: ``` $ vessel new myproject $ cd myproject $ bundle install $ vessel generate example.com $ vessel start example.com ``` ### Project skeleton Here's the structure that every new project comes with: ``` myproject ├── Gemfile ├── config │ ├── boot.rb │ ├── environments │ │ ├── dev/dev.rb │ │ └── prod/prod.rb │ ├── fields │ └── middleware ├── crawlers ├── lib │ ├── helpers │ └── loader.rb └── log ``` One crawler per site in `crawlers/`, shared middleware, and field types in `config/`. ### Debugging The debugging command `vessel parse` can fetch one URL and runs a specific handler against it: ``` $ vessel parse example.com https://example.com/products/1 parse_product ``` It's Vessel's answer to `scrapy parse`, and much faster than re-running a whole crawl to test a selector change. ## Example projects ### Mapping internal links To test Vessel 0.3 on something real, I wrote a [crosslink mapper](https://github.com/serpapi/vessel-crosslink-mapper). You can point it at a site section like `serpapi.com/blog` and it crawls every page in scope, records which pages link to which, and reports on the site's internal linking. The whole thing is one file plus a Gemfile. The crawler itself is short. It uses the Mechanize driver (a blog does not need Chrome), normalizes URLs so `/blog` and `/blog/` count as one page, stays inside the start URL's host and path prefix, and yields one item per page: ```ruby class CrosslinkMapper < ApplicationCrawler domain "serpapi.com" start_urls AppSettings::START_URL driver :mechanize network_error_attempts 2 def parse current = Urls.normalize(url) outlinks = Set.new css("a[href]").each do |a| target = Urls.normalize(absolute_url(a[:href])) outlinks << target if target && Urls.in_scope?(target) && target != current end field :url, value: current field :title, value: at_css("title")&.text # stripped by the :title FieldType field :outlinks, value: outlinks.to_a.sort yield fields outlinks.each do |link| yield request(url: link, handler: :parse) if self.class.claim_slot?(link) end end def on_error(request, error) LinkGraph.record_failure(Urls.normalize(request&.url), error.message) end def after(_stats) LinkGraph.write_reports(start_url: AppSettings::START_URL, budget: AppSettings::MAX_PAGES) end end ``` Vessel's defaults do a lot of the work here. Deduplication is free (`once: true`), so yielding every outlink back to `parse` is safe; `claim_slot?` only adds a page budget on top. Retries are free too. And `on_error` doubles as a broken-link detector: the Mechanize driver raises on a 404 or a 500 instead of handing you the page, so every URL that lands there after its retries is a dead internal link, recorded together with the pages that link to it. A `CollectPage` middleware feeds every yielded item into an in-memory link graph, a `FieldType` strips every `:title` in one declaration, and the environments differ the way the skeleton suggests. The `dev` environment runs two threads with an extra `Debug` middleware that echoes each page, prod runs four threads and logs to a file. Here are the relevant bits: ```ruby # config/middleware/collect_page.rb class CollectPage < Vessel::Middleware def call(hash, _fields) LinkGraph.add_page(hash) hash end end # config/environments/dev/dev.rb class ApplicationCrawler < Vessel::Cargo threads max: 2 middleware "Debug", "CollectPage" end # config/environments/prod/prod.rb class ApplicationCrawler < Vessel::Cargo threads max: 4 middleware "CollectPage" end ``` After the crawl, the script writes a `report/` directory: every page sorted by inlinks, every page sorted by outlinks, orphan pages nothing links to, broken links with their sources, and the uncrawled frontier. It also exports the full graph as JSON and as a Graphviz file, so `sfdp -Tsvg crosslinks.dot -o crosslinks.svg` draws the link map. A capped 15-page test run against serpapi.com/blog found 93 internal links, with the blog index at 14 inlinks and author pages dominating the top of the list, which is what you would expect from a Ghost blog. The same numbers for your own site will be more interesting. Orphan pages are posts your readers cannot find, and `broken-links.txt` is a list to fix. ### Screenshotting a website with Chrome The crosslink mapper never needed a browser so I also build a [screenshot archiver](https://github.com/serpapi/vessel-screenshot-archiver) that shows you how Vessel works with Chrome. Point it at a site and it renders every page in headless Chrome, saves a full-page PNG per page, and generates an `index.html` contact sheet, a grid of every page's screenshot with its title and link. Useful as a visual archive before a redesign, or for seeing a whole site at once. The crawling skeleton is the same as before: ```ruby class ScreenshotArchiver < ApplicationCrawler domain "serpapi.com" start_urls AppSettings::START_URL network_error_attempts 2 # Chrome never loads trackers and widgets, so pages render faster # and consent banners stay out of the screenshots. blacklist [/googletagmanager/, /google-analytics/, /doubleclick/, /hotjar/, /intercom/] def parse current = Urls.normalize(url) # `page` is the raw Ferrum::Page. Scroll to the bottom so # lazy-loaded images render, then back up for the capture. page.execute("window.scrollTo(0, document.body.scrollHeight)") sleep 0.5 page.execute("window.scrollTo(0, 0)") file = File.join(AppSettings::SHOTS_DIR, "#{Urls.slug(current)}.png") begin page.screenshot(path: file, full: true) rescue Ferrum::Error # Chrome refuses to capture extremely tall pages as one bitmap. page.screenshot(path: file) end field :url, value: current field :title, value: page.evaluate("document.title") field :file, value: file yield fields each_in_scope_link do |link| yield request(url: link, handler: :parse) if self.class.claim_slot?(link) end end end ``` Above, `page` gives the handler the raw `Ferrum::Page`, so it can execute JavaScript, scroll, and call `page.screenshot(full: true)`. `blacklist` tells Chrome which resources never to load. And the `driver` options (`headless`, `window_size`, `timeout`) go straight to `Ferrum::Browser.new`. I ran it against [SerpApi](https://serpapi.com)'s API feature pages like `/google-events-api`, `/bing-search-api`, and so on. They share no path prefix, so the script scopes by a regex instead of a path. I noticed Chrome cannot capture extremely tall pages as a single bitmap as it hits a texture size limit, so the script falls back to a viewport shot when the full-page capture fails. Also, the Ferrum driver hands you error pages instead of raising so you might end up screenshotting the 404 pages. ### Production considerations Generated crawlers inherit from `ApplicationCrawler`, which is defined per environment and selected by `VESSEL_ENV` (defaulting to `dev`). Settings are inherited and deep-copied into subclasses, so the natural setup is: - headful Chrome, one thread, and verbose logging in development (`dev.rb`) - headless, full thread pool, proxies, and delays in production (`prod.rb`) ## Upgrading from Vessel 0.2 Here are four breaking changes for those coming from v0.2\. If you have an old Vessel crawler running somewhere, you'll need to look into these: - `Middleware` is now a class with `call(hash, fields)`, declared by name: `middleware "Debug", "Save"`. The old `Middleware.build` chain is gone. - `timeout`and`ferrum`settings are replaced by `driver`: `driver :ferrum, timeout: 30`. - `intercept` is replaced by `blacklist` and `whitelist`. - URLs are visited **only once** by default. If your crawler relied on revisiting pages, pass `once: false` on those requests. ### Ruby support The minimum Ruby version is now 3.1 and the dependencies were brought up to date, including explicitly declaring `logger` to also load correctly on Ruby 4.0, where logger is no longer a default gem. ## When to use Vessel (and when not to) Use Vessel when: - You are crawling many pages of one or a few sites and want structure instead of a hand-rolled queue - The site renders with JavaScript and you would need a browser anyway - You have several crawlers and want shared middleware, field normalization, and environments - You are a Ruby shop and the alternative was "rewrite it in Scrapy" Skip it when: - You are scraping a handful of static pages... Faraday plus Nokogiri is simpler (see the [Ruby web scraping guide](https://serpapi.com/blog/ruby-web-scraping)), - You need to crawl millions of pages or run a distributed crawl, for which Scrapy is still better - You need robots.txt compliance or request-level middleware out of the box ## Conclusion For years, we had to build our Ruby crawling frameworks ourselves or accept the fact to use Scrapy from [Python](https://serpapi.com/blog/python-web-scraping-tutorial/). Vessel 0.3 changes that now as it's probably able to run most of the typical crawling jobs out there. Scraping a few thousand pages of a single site with a real browser is now really easy in the Ruby land as well.