Vessel 0.3: Ruby finally gets its Scrapy crawling framework

Python developers have Scrapy, one of the best tool for web scraping out there. And us Rubyists? We have to piece individual pieces together. Nokogiri, Ferrum, Mechanize, Faraday... but no framework that ties fetching, parsing, concurrency, deduplication, retries, and data pipelines together. Vessel was supposed to be that framework until it went a little quiet.

Now it is back. Vessel 0.3 is the first release in four years, and it is not a maintenance bump. The internals were rewritten and the feature list finally reads like a real crawling framework. We got pluggable drivers (real Chrome or plain HTTP), a fields API, a middleware pipeline, proxy rotation, cookies, retries, callbacks, and a CLI that generates whole projects.

What is a crawling framework?

A crawling framework bundles scraping and web crawling concerns into one cohesive experience. Tthe framework schedules requests, fetches pages concurrently, deduplicates URLs, retries failures, and pushes whatever you extract through a processing pipeline. In other words, an HTTP client fetches a page, a crawler follows links, but a crawling framework runs the whole loop from scheduling crawls, fetching pages, parsing sources, and storing data.

Vessel is a high-level web crawling framework for Ruby. You only really subclass Vessel::Cargo, declare a domain and start URLs, and write handler methods that extract data and yield new requests. Scheduling, concurrency, visiting every URL only once, retrying network errors, and pushing extracted items through a processing pipeline is all done for you by Vessel. This is a stark contrast to simple libraries that handle only some of these concerns and makes the user to stitch everything together.

Scraping a page vs. crawling a site

Scraping one page is easy in Ruby. Fetch the HTML, parse it with Nokogiri, done. I covered all of that in my complete guide to web scraping with Ruby. Crawling a site is a different problem. You are not extracting data from a page, you are extracting data from a graph of pages, and suddenly you need:

• A queue of URLs to visit
• Deduplication so you do not visit the same URL twice
• Concurrency so the crawl finishes this week
• Retries for pages that time out
• Rate limiting to respect a given site limits

You can hand-roll all of this with a Queue, a Set, and a thread pool. Many of you likely have. But this is exactly the kind of plumbing a framework like Scrapy offered for years.

Where Spidr and Anemone fall short

Ruby does have crawling gems. Don't worry, I know. Spidr and Anemone will happily walk every link on a site and hand you each page. They are fine for link checking or building a URL inventory. I used Spidr before for this kind of tasks.

But they are crawlers, not crawling frameworks. They give you pages but lacks extraction structure, data pipelines, JavaScript rendering, proxies, and per-request state. Anemone has not seen a release in years, and neither runs JavaScript, so any site that renders content client-side is out of reach.

How Vessel compares to Scrapy

Vessel is not a 1:1 copy of Scrapy but there are similarities. We could say the core mental model is now genuinely the same:

And in two places Vessel ships more than Scrapy does out of the box:

• A real browser by default. Scrapy is HTTP-first, and JavaScript rendering means bolting on scrapy-playwright or Splash. Vessel's default driver is actual Chrome via Ferrum, with a plain-HTTP Mechanize driver when you do not need it. Vessel is a browser-first.
• Built-in proxy rotation. Round-robin and shuffled rotation ship with the gem while in Scrapy that is a third-party middleware.

However, Vessel isn't a Scrapy 1:1 replacement even if they share the programming model. Scrapy's scale and ecosystem is still better:

• There is no downloader middleware layer. Vessel's middleware is an item pipeline only. There is no global hook to rewrite requests and responses, which is Scrapy's killer extension point.
• There are no feed exports. Nothing built-in dumps items to CSV, JSON, or S3 — you write a Save middleware yourself.
• There is no HTTP cache, no AutoThrottle, and no per-domain concurrency controls. Vessel's delay only applies when the crawler runs single-threaded.
• There is no robots.txt handling.
• The concurrency model is a thread pool, one Chrome page per core by default. Scrapy's async engine comfortably crawls millions of pages while Vessel is built for hundreds to thousands.
• Scrapy has fifteen years of plugins, docs, scrapyd, and scrapy-redis behind it. Vessel tracks its first release to 2021.

For most Ruby scraping jobs, the model is what was missing, and Vessel now has it. But if you are running a big Scrapy cluster, you might still need to keep it.

A first crawler in 30 lines

Here is the canonical example, crawling quotes.toscrape.com with pagination:

require "json"
require "vessel"

class QuotesToScrapeCom < Vessel::Cargo
  domain "quotes.toscrape.com"
  start_urls "https://quotes.toscrape.com/tag/humor/"

  def parse
    css("div.quote").each do |quote|
      yield({
        author: quote.at_xpath("span/small").text,
        text: quote.at_css("span.text").text
      })
    end

    next_page = at_xpath("//li[@class='next']/a[@href]")
    return unless next_page

    yield request(url: absolute_url(next_page[:href]), handler: :parse)
  end
end

quotes = []
QuotesToScrapeCom.run { |q| quotes << q }
puts JSON.generate(quotes)

Save it as quotes.rb, run bundle exec ruby quotes.rb > quotes.json, and you have every humor quote across all pages.

Vessel visits the start URL and calls the handler (parse by default). Inside a handler you query the page directly with css, at_css, xpath, and at_xpath. Yielding a hash emits an item while yielding a request schedules another page, handled concurrently by a thread pool sized to your cores.

The same URL is never visited twice, and pagination is just a handler yielding a request back to itself.

What's new in 0.3

Pluggable and custom drivers

Pluggable drivers give you Chrome when you need it, plain HTTP when you don't. Pages are now fetched by a pluggable driver:

class MyScraper < Vessel::Cargo
  driver :ferrum, headless: true, timeout: 30
  # or
  driver :mechanize
end

:ferrum is the default: a real Chrome, JavaScript and all, with sensible crawling defaults (certificate errors ignored, JS errors swallowed, generous timeouts). :mechanize is plain HTTP so no browser process, no JavaScript, much faster and lighter.

Custom drivers are supported too. Subclass Vessel::Driver, implement start, stop, and create_page, and register it.

One Ferrum-only nicety: blacklist and whitelist patterns control which resources Chrome loads, so you can skip images, fonts, and trackers:

class MyScraper < Vessel::Cargo
  blacklist [/\.png$/, /googletagmanager/]
end

This can nicely speed up the overall crawling.

The fields API with normalization

Instead of assembling hashes by hand, handlers can declare fields that gets automatically normalized:

def parse
  field :author, value: at_xpath("span/small").text
  field :text, value: at_css("span.text").text
  field :html, value: nil, service: true do
    raw
  end

  yield fields
end

service: true keeps a field out of the resulting item but available to the middleware, handy for carrying the raw HTML along for debugging. Fields can be renamed, and FieldType normalizes fields by name across all crawlers in one place:

Vessel::Cargo::FieldType.add(:price) { |value| value.to_s.gsub(/[^\d.]/, "").to_f }

Every field :price in every crawler now comes out as a Float. If you have ever maintained five scrapers that each clean prices slightly differently, you know why this exists. It is a lighter take on Scrapy's item loaders and processors.

Middleware as a pipeline

Everything a handler yields that is not a request goes through the middleware pipeline, which runs in its own thread pool. A middleware is a class with a call(hash, fields) method:

class Sanitize < Vessel::Middleware
  def call(hash, fields)
    hash.transform_values { |v| v.is_a?(String) ? v.strip : v }
  end
end

class Save < Vessel::Middleware
  def call(hash, fields)
    raise Vessel::Middleware::InvalidItemError if hash[:text].to_s.empty?

    DB[:quotes].insert(hash)
    hash
  end
end

class MyScraper < Vessel::Cargo
  middleware "Sanitize", "Save"
end

Each middleware receives the hash from the previous one plus the original fields object. Raising InvalidItemError silently drops an item. This is Scrapy's item pipeline, down to the drop-item semantics. Validation, cleaning, and persistence live here instead of being tangled into your parse handlers.

For quick scripts, a block passed to .run replaces the whole pipeline, as in the first example.

Proxy rotation and cookies

Proxy rotation is built in. Subclass Vessel::RoundRobinProxy or Vessel::ShuffledProxy, define a PROXIES constant, and the driver takes the next proxy for every page it creates:

class MyProxy < Vessel::ShuffledProxy
  PROXIES = [
    { host: "127.0.0.1", port: 8080, user: "user1", password: "password1" },
    { host: "127.0.0.1", port: 8081, user: "user2", password: "password2" }
  ].freeze
end

class MyScraper < Vessel::Cargo
  proxy MyProxy
end

Cookies got a proper API as well: cookie and cookies set them up front, cookies received from responses are kept for subsequent requests by default, and allow_cookies false turns that off. Headers, cookies, and delays can also be overridden per request, and requests carry an arbitrary data hash over to the response:

def parse
  yield request(url: "/page/2/", handler: :parse_page, data: { category: "humor" })
end

def parse_page
  puts response.data[:category] # => "humor"
end

That data hash is Scrapy's meta, and it solves the classic crawling problem of carrying context (which category page did this product come from?) across requests.

Retries, dedup, and callbacks

A request that fails with a network error (timeout, socket error, bad status) is retried, five times by default, with the browser restarted between attempts:

class MyScraper < Vessel::Cargo
  network_error_attempts 5
end

When the attempts run out, the error lands in the on_error(request, error) callback, one of a set of new lifecycle hooks (before_start, before, after_change, info, after, before_stop). The info callback fires every few seconds with a stats hash (requests enqueued, items processed, items rejected), which is exactly what you want to log in a long crawl.

Deduplication is now on by default: the same URL is not visited twice unless you pass once: false. And 0.3 fixed a subtle race where threads competing for the same URL could visit it in parallel.

The new Vessel CLI

Vessel comes with a CLI to list crawlers, inspect their settings, and run them. To define a crawler, a single file is usually fine for one. For a collection of them, Vessel now generates a small project for you:

$ vessel new myproject
$ cd myproject
$ bundle install
$ vessel generate example.com
$ vessel start example.com

Project skeleton

Here's the structure that every new project comes with:

myproject
├── Gemfile
├── config
│   ├── boot.rb
│   ├── environments
│   │   ├── dev/dev.rb
│   │   └── prod/prod.rb
│   ├── fields
│   └── middleware
├── crawlers
├── lib
│   ├── helpers
│   └── loader.rb
└── log

One crawler per site in crawlers/, shared middleware, and field types in config/.

Debugging

The debugging command vessel parse can fetch one URL and runs a specific handler against it:

$ vessel parse example.com https://example.com/products/1 parse_product

It's Vessel's answer to scrapy parse, and much faster than re-running a whole crawl to test a selector change.

Example projects

Mapping internal links

To test Vessel 0.3 on something real, I wrote a crosslink mapper. You can point it at a site section like serpapi.com/blog and it crawls every page in scope, records which pages link to which, and reports on the site's internal linking. The whole thing is one file plus a Gemfile.

The crawler itself is short. It uses the Mechanize driver (a blog does not need Chrome), normalizes URLs so /blog and /blog/ count as one page, stays inside the start URL's host and path prefix, and yields one item per page:

class CrosslinkMapper < ApplicationCrawler
  domain "serpapi.com"
  start_urls AppSettings::START_URL
  driver :mechanize
  network_error_attempts 2

  def parse
    current = Urls.normalize(url)
    outlinks = Set.new

    css("a[href]").each do |a|
      target = Urls.normalize(absolute_url(a[:href]))
      outlinks << target if target && Urls.in_scope?(target) && target != current
    end

    field :url, value: current
    field :title, value: at_css("title")&.text # stripped by the :title FieldType
    field :outlinks, value: outlinks.to_a.sort
    yield fields

    outlinks.each do |link|
      yield request(url: link, handler: :parse) if self.class.claim_slot?(link)
    end
  end

  def on_error(request, error)
    LinkGraph.record_failure(Urls.normalize(request&.url), error.message)
  end

  def after(_stats)
    LinkGraph.write_reports(start_url: AppSettings::START_URL, budget: AppSettings::MAX_PAGES)
  end
end

Vessel's defaults do a lot of the work here. Deduplication is free (once: true), so yielding every outlink back to parse is safe; claim_slot? only adds a page budget on top. Retries are free too. And on_error doubles as a broken-link detector: the Mechanize driver raises on a 404 or a 500 instead of handing you the page, so every URL that lands there after its retries is a dead internal link, recorded together with the pages that link to it.

A CollectPage middleware feeds every yielded item into an in-memory link graph, a FieldType strips every :title in one declaration, and the environments differ the way the skeleton suggests. The dev environment runs two threads with an extra Debug middleware that echoes each page, prod runs four threads and logs to a file. Here are the relevant bits:

# config/middleware/collect_page.rb
class CollectPage < Vessel::Middleware
  def call(hash, _fields)
    LinkGraph.add_page(hash)
    hash
  end
end

# config/environments/dev/dev.rb
class ApplicationCrawler < Vessel::Cargo
  threads max: 2
  middleware "Debug", "CollectPage"
end

# config/environments/prod/prod.rb
class ApplicationCrawler < Vessel::Cargo
  threads max: 4
  middleware "CollectPage"
end

After the crawl, the script writes a report/ directory: every page sorted by inlinks, every page sorted by outlinks, orphan pages nothing links to, broken links with their sources, and the uncrawled frontier. It also exports the full graph as JSON and as a Graphviz file, so sfdp -Tsvg crosslinks.dot -o crosslinks.svg draws the link map.

A capped 15-page test run against serpapi.com/blog found 93 internal links, with the blog index at 14 inlinks and author pages dominating the top of the list, which is what you would expect from a Ghost blog. The same numbers for your own site will be more interesting. Orphan pages are posts your readers cannot find, and broken-links.txt is a list to fix.

Screenshotting a website with Chrome

The crosslink mapper never needed a browser so I also build a screenshot archiver that shows you how Vessel works with Chrome. Point it at a site and it renders every page in headless Chrome, saves a full-page PNG per page, and generates an index.html contact sheet, a grid of every page's screenshot with its title and link. Useful as a visual archive before a redesign, or for seeing a whole site at once.

The crawling skeleton is the same as before:

class ScreenshotArchiver < ApplicationCrawler
  domain "serpapi.com"
  start_urls AppSettings::START_URL
  network_error_attempts 2

  # Chrome never loads trackers and widgets, so pages render faster
  # and consent banners stay out of the screenshots.
  blacklist [/googletagmanager/, /google-analytics/, /doubleclick/, /hotjar/, /intercom/]

  def parse
    current = Urls.normalize(url)

    # `page` is the raw Ferrum::Page. Scroll to the bottom so
    # lazy-loaded images render, then back up for the capture.
    page.execute("window.scrollTo(0, document.body.scrollHeight)")
    sleep 0.5
    page.execute("window.scrollTo(0, 0)")

    file = File.join(AppSettings::SHOTS_DIR, "#{Urls.slug(current)}.png")
    begin
      page.screenshot(path: file, full: true)
    rescue Ferrum::Error
      # Chrome refuses to capture extremely tall pages as one bitmap.
      page.screenshot(path: file)
    end

    field :url, value: current
    field :title, value: page.evaluate("document.title")
    field :file, value: file
    yield fields

    each_in_scope_link do |link|
      yield request(url: link, handler: :parse) if self.class.claim_slot?(link)
    end
  end
end

Above, page gives the handler the raw Ferrum::Page, so it can execute JavaScript, scroll, and call page.screenshot(full: true). blacklist tells Chrome which resources never to load. And the driver options (headless, window_size, timeout) go straight to Ferrum::Browser.new.

I ran it against SerpApi's API feature pages like /google-events-api, /bing-search-api, and so on. They share no path prefix, so the script scopes by a regex instead of a path. I noticed Chrome cannot capture extremely tall pages as a single bitmap as it hits a texture size limit, so the script falls back to a viewport shot when the full-page capture fails. Also, the Ferrum driver hands you error pages instead of raising so you might end up screenshotting the 404 pages.

Production considerations

Generated crawlers inherit from ApplicationCrawler, which is defined per environment and selected by VESSEL_ENV (defaulting to dev). Settings are inherited and deep-copied into subclasses, so the natural setup is:

• headful Chrome, one thread, and verbose logging in development (dev.rb)
• headless, full thread pool, proxies, and delays in production (prod.rb)

Upgrading from Vessel 0.2

Here are four breaking changes for those coming from v0.2. If you have an old Vessel crawler running somewhere, you'll need to look into these:

• Middleware is now a class with call(hash, fields), declared by name: middleware "Debug", "Save". The old Middleware.build chain is gone.
• timeout and ferrum settings are replaced by driver: driver :ferrum, timeout: 30.
• intercept is replaced by blacklist and whitelist.
• URLs are visited only once by default. If your crawler relied on revisiting pages, pass once: false on those requests.

Ruby support

The minimum Ruby version is now 3.1 and the dependencies were brought up to date, including explicitly declaring logger to also load correctly on Ruby 4.0, where logger is no longer a default gem.

When to use Vessel (and when not to)

Use Vessel when:

• You are crawling many pages of one or a few sites and want structure instead of a hand-rolled queue
• The site renders with JavaScript and you would need a browser anyway
• You have several crawlers and want shared middleware, field normalization, and environments
• You are a Ruby shop and the alternative was "rewrite it in Scrapy"

Skip it when:

• You are scraping a handful of static pages... Faraday plus Nokogiri is simpler (see the Ruby web scraping guide),
• You need to crawl millions of pages or run a distributed crawl, for which Scrapy is still better
• You need robots.txt compliance or request-level middleware out of the box

Conclusion

For years, we had to build our Ruby crawling frameworks ourselves or accept the fact to use Scrapy from Python. Vessel 0.3 changes that now as it's probably able to run most of the typical crawling jobs out there. Scraping a few thousand pages of a single site with a real browser is now really easy in the Ruby land as well.