Benchmarking Ruby 3.1 (+YJIT) vs Ruby 2.7

Ruby 3 has been released for a while. As a core language in SerpApi, the new version is definitely going to be tried out, especially given that Ruby 2.6 has already reached end-of-life, and it will soon come to Ruby 2.7, which SerpApi is currently running on. Apart from the new language features introduced in Ruby 3, performance is our main concern. Although there have been many comparisons between Ruby 3.1 and Ruby 2.7, we want to test them with our own environment to see if Ruby 3 will have a performance improvement on SerpApi.

Ruby 3.1 has shipped with a new JIT compiler, YJIT. It claims to have performance improvements on most real-world software, up to 22% on railsbench and 39% on liquid-render. We will be comparing Ruby 2.7.2, Ruby 3.1.2 and Ruby 3.1.2 + YJIT.

All the following benchmarks were run on a DigitalOcean CPU-Optimized, 2 vCPUs, 4 GB virtual machine.

Benchmark on yjit-bench

Our first benchmark ran on yjit-bench, provided by Shopify, the developer of YJIT.

Run the Ruby 3.1.2 and Ruby 3.1.2 + YJIT benchmark by switching to Ruby 3.1.2 and running ./run_benchmarks.rb

Run the Ruby 2.7.2 benchmark by switching to Ruby 2.7.2 and running ./run_benchmarks.rb -e ruby

Here are the results. Lower is better.

Raw data:

The results showed that Ruby 3.1.2 is around the same performance as Ruby 2.7.2, as reported by other benchmarks. YJIT turned out to outperform the other two a lot, which is quite impressive. railsbench, that we paid the most attention to, was improved by 23.3% in this test. These results gave us the confidence to carry on to the next benchmark on SerpApi.

Benchmark on SerpApi

We benchmarked the most time-consuming process of SerpApi: HTML parsing and JSON output construction. A static google search result page was fed for the tests. Both time and memory usage were recorded. The following script was used for benchmarking.

def do_parse
  Search.new(args_of_a_google_result_page).parse!
end

10.times {
  puts Benchmark.measure {
    30.times { do_parse }
  }
}

require 'memory_profiler'
MemoryProfiler.report { do_parse }.pretty_print

Here are the results of the time elapsed. Lower is better.

Raw data:

Ruby 2.7.2
---------------
25.047808   0.161161  25.208969 ( 25.268428)
23.927413   0.035128  23.962541 ( 23.980549)
24.188614   0.031934  24.220548 ( 24.257346)
28.051661   0.003824  28.055485 ( 28.117897)
25.007042   0.039772  25.046814 ( 25.066986)
24.256959   0.023895  24.280854 ( 24.298495)
23.700391   0.000000  23.700391 ( 23.718018)
24.739665   0.019597  24.759262 ( 24.777065)
23.896913   0.024010  23.920923 ( 23.941751)
24.410275   0.019975  24.430250 ( 24.458073)

Ruby 3.1.2
---------------
23.831084   0.142897  23.973981 ( 24.013818)
25.364707   0.035620  25.400327 ( 25.435810)
26.620177   0.020064  26.640241 ( 26.657792)
27.575943   0.024074  27.600017 ( 27.619430)
27.609700   0.015994  27.625694 ( 27.645148)
27.890730   0.011946  27.902676 ( 27.921206)
28.092639   0.015964  28.108603 ( 28.127349)
27.807529   0.035993  27.843522 ( 27.862086)
27.865685   0.047982  27.913667 ( 27.933122)
27.976437   0.031967  28.008404 ( 28.027206)

Ruby 3.1.2 + YJIT
---------------
23.583271   1.066498  24.649769 ( 24.656726)
25.698567   0.884720  26.583287 ( 26.593125)
27.064727   0.955691  28.020418 ( 28.039999)
27.833003   0.923952  28.756955 ( 28.777363)
27.415354   0.963869  28.379223 ( 28.395398)
28.977574   0.945236  29.922810 ( 29.962006)
27.541010   0.947636  28.488646 ( 28.508437)
28.373851   0.899978  29.273829 ( 29.293450)
27.706739   0.983724  28.690463 ( 28.705540)
28.299704   0.907967  29.207671 ( 29.227521)

YJIT, the absolute winner of the previous benchmark, was not performing as expected, even worse than disabled. There are frustrating results.

However, It's interesting that the first output of Ruby 3.1.2 and Ruby 3.1.2 + YJIT were rather low, then the numbers grew gradually. Rather than "warming up", it performed like "cooling down". Some extra efforts are needed to figure out why Ruby 3.1.2 will have this kind of behavior on SerpApi.

The user time of Ruby 3.1.2 and Ruby 3.1.2 + YJIT (the first column) were very close. However, the latter took much more time on sys time (the second column), resulting in a slower time overall.

We guess that the reason why SerpApi was not improved by YJIT is that Nokogiri, the HTML parser we are using, is mostly written in C extension. JIT compilers cannot optimize C code. And there may be extra overhead calling C code from ruby when JIT compiler is enabled.

We are also attaching memory consumption profiled by memory_profiler.

We've run twice for each ruby version because the first run seems to allocate much more memory than the rest. But the overall results were the same. Ruby 2.7.2 allocated the least memory, followed by Ruby 3.1.2 + YJIT, Ruby 3.1.2 consumed the most.

Conclusion

It can be concluded the new version of Ruby was not performing well on SerpApi. We are sensitive to performance, and it seems we might not migrate to Ruby 3 for now. But these are just early results of the whole benchmark schedule. We will also compare Rails 6 vs Rails 7 and other new versions of libraries to make the final decision.

Thanks for reading, see you in the next benchmark blog post.