How Rspec Reporter improves our daily productivity

SerpApi is powered by Ruby on Rails and its ecosystem. For sure, we use Rspec as a test framework. Like all Ruby on Rails projects, we have unit tests, request tests and integration tests. 

We care about test suites and believe that test suites will help us to move fast and not break anything.

SerpApi provides APIs for certain engines like Google, Bing, Yahoo, and Yandex… Given parameters like query, location, and pagination info, we will request Search Engines to get HTML layouts and return JSON results. So, our testing process follows the same process: we test massive layouts from Search Engines and make requests for integration tests.

Our typical integration spec looks like this:

  describe "Organic Results for coffeee wiki espresso", type: :request do
    before :all do
      get search_path q: "coffeee wiki espresso", location: "Dallas, Texas, United States", hl: "en", gl: "us"
      @json = response.parsed_body
    end

    it "returns http success" do
      response.code.should eq "200"
    end

    it "contains organic results array" do
      @json["organic_results"].should be_an(Array)
    end

    it "total_results" do
      @json["search_information"]["total_results"].should be_an(Integer)
      @json["search_information"]["total_results"].should be > 1000000
    end

    it "time_taken_displayed" do
      @json["search_information"]["time_taken_displayed"].should be_an(Float)
    end

    describe "organic_results_state" do
      specify { @json["search_information"]["organic_results_state"].should eq 'Some results for exact spelling but showing fixed spelling' }
    end

    # And some other specs related to organic results
    
  end

Search Engines introduce new layouts and new results every day. Some new results are beta testing so sometimes we can catch the new HTML layouts. We do make real requests to Search Engines in our integration specs and sometimes it’s failed cause these new HTML layouts.

We WANT to store the new HTML layouts for the troubleshooting process. But where do we store the HTML and how do we do it?

The most convenient way for us is to display more information when the spec fails. Instead of simple red dots and some failed expectations, we want to upload a new HTML layout to S3 and show the link—an extra line for the spec report. So when the specs fail on CI or locally, we still open the HTML via the S3 link.

Rspec reporter

To customize the Rspec report, we can take a look at the Rspec report module: https://github.com/rspec/rspec-core/blob/main/lib/rspec/core/reporter.rb

Rspec fires notifications for many events, including when specs are failed. That is a great starting point for our customization. To understand how it works, you can take a look at this small built-in formatter:

https://github.com/rspec/rspec-core/blob/main/lib/rspec/core/formatters/progress_formatter.rb

Another way to “listen” to all notifications is to `register_listener` directly.

https://github.com/rspec/rspec-core/blob/main/lib/rspec/core/reporter.rb#L37

In `example_failed(notification)`, you have all the data of the current example, like parameters, file path, and report format. With some great Ruby metaprogramming functions, you can access all available instance variables on running specs and get the job done.

We're quite happy with our setup, the CI runs anytime and keeps unexpected HTML layouts for us.

We love Ruby and are still hiring.