Benchmarking Ruby 3.1 (+YJIT) vs Ruby 2.7

Ruby 3 has been released for a while. As a core language in SerpApi, the new version is definitely going to be tried out, especially given that Ruby 2.6 has already reached end-of-life, and it will soon come to Ruby 2.7, which SerpApi is currently running on. Apart from the new language features introduced in Ruby 3, performance is our main concern. Although there have been many comparisons between Ruby 3.1 and Ruby 2.7, we want to test them with our own environment to see if Ruby 3 will have a performance improvement on SerpApi.

Ruby 3.1 has shipped with a new JIT compiler, YJIT. It claims to have performance improvements on most real-world software, up to 22% on railsbench and 39% on liquid-render. We will be comparing Ruby 2.7.2, Ruby 3.1.2 and Ruby 3.1.2 + YJIT.

All the following benchmarks were run on a DigitalOcean CPU-Optimized, 2 vCPUs, 4 GB virtual machine.

Benchmark on yjit-bench

Our first benchmark ran on yjit-bench, provided by Shopify, the developer of YJIT.

Run the Ruby 3.1.2 and Ruby 3.1.2 + YJIT benchmark by switching to Ruby 3.1.2 and running ./run_benchmarks.rb

Run the Ruby 2.7.2 benchmark by switching to Ruby 2.7.2 and running ./run_benchmarks.rb -e ruby

Here are the results. Lower is better.

Raw data:

benchmark ruby2.7.2 ruby3.1.2 ruby3.1.2+yjit
30k_ifelse 2835.369178 2871.740415 347.6600132
30k_methods 7668.80311 7558.231579 825.0258768
activerecord 194.1947169 192.832139 134.1704337
binarytrees 344.994329 442.7235988 331.8990998
cfunc_itself 87.15214192 98.89735636 45.6197738
chunky_png 858.1087336 970.3283633 671.3995022
erubi 452.75688 474.8846801 407.0801165
erubi_rails 34.6515834 35.59898836 25.86827893
etanni 525.0555173 604.5594306 843.4775177
fannkuchredux 4544.754985 6215.255843 6271.386797
fib 190.5258242 220.4583379 57.74113797
getivar 105.105136 92.75804685 39.73911569
hexapdf 3403.161764 3296.487006 2545.836672
keyword_args 228.9924551 271.8941905 52.07178436
lee 1078.889968 1145.596831 839.7007101
liquid-render 181.0265133 205.334054 133.918901
mail 168.2398866 181.6809455 167.7362736
nbody 118.3354334 110.2136637 83.34042853
optcarrot 5182.830082 5723.337052 3678.027454
psych-load 2598.436543 2434.784967 1932.677771
railsbench 3825.04206 3505.656024 2687.687757
respond_to 238.2646554 248.3746187 184.7939404
ruby-lsp 105.0971961 130.3235934
rubykon 10704.44953 12166.72257 6502.561311
setivar 67.84215721 69.57942243 47.23990744
str_concat 121.3709915 131.0851263 130.3337132

The results showed that Ruby 3.1.2 is around the same performance as Ruby 2.7.2, as reported by other benchmarks. YJIT turned out to outperform the other two a lot, which is quite impressive. railsbench, that we paid the most attention to, was improved by 23.3% in this test. These results gave us the confidence to carry on to the next benchmark on SerpApi.

Benchmark on SerpApi

We benchmarked the most time-consuming process of SerpApi: HTML parsing and JSON output construction. A static google search result page was fed for the tests. Both time and memory usage were recorded. The following script was used for benchmarking.

def do_parse
  Search.new(args_of_a_google_result_page).parse!
end

10.times {
  puts Benchmark.measure {
    30.times { do_parse }
  }
}

require 'memory_profiler'
MemoryProfiler.report { do_parse }.pretty_print

Here are the results of the time elapsed. Lower is better.

Raw data:

Ruby 2.7.2
---------------
25.047808   0.161161  25.208969 ( 25.268428)
23.927413   0.035128  23.962541 ( 23.980549)
24.188614   0.031934  24.220548 ( 24.257346)
28.051661   0.003824  28.055485 ( 28.117897)
25.007042   0.039772  25.046814 ( 25.066986)
24.256959   0.023895  24.280854 ( 24.298495)
23.700391   0.000000  23.700391 ( 23.718018)
24.739665   0.019597  24.759262 ( 24.777065)
23.896913   0.024010  23.920923 ( 23.941751)
24.410275   0.019975  24.430250 ( 24.458073)
Ruby 3.1.2
---------------
23.831084   0.142897  23.973981 ( 24.013818)
25.364707   0.035620  25.400327 ( 25.435810)
26.620177   0.020064  26.640241 ( 26.657792)
27.575943   0.024074  27.600017 ( 27.619430)
27.609700   0.015994  27.625694 ( 27.645148)
27.890730   0.011946  27.902676 ( 27.921206)
28.092639   0.015964  28.108603 ( 28.127349)
27.807529   0.035993  27.843522 ( 27.862086)
27.865685   0.047982  27.913667 ( 27.933122)
27.976437   0.031967  28.008404 ( 28.027206)
Ruby 3.1.2 + YJIT
---------------
23.583271   1.066498  24.649769 ( 24.656726)
25.698567   0.884720  26.583287 ( 26.593125)
27.064727   0.955691  28.020418 ( 28.039999)
27.833003   0.923952  28.756955 ( 28.777363)
27.415354   0.963869  28.379223 ( 28.395398)
28.977574   0.945236  29.922810 ( 29.962006)
27.541010   0.947636  28.488646 ( 28.508437)
28.373851   0.899978  29.273829 ( 29.293450)
27.706739   0.983724  28.690463 ( 28.705540)
28.299704   0.907967  29.207671 ( 29.227521)

YJIT, the absolute winner of the previous benchmark, was not performing as expected, even worse than disabled. There are frustrating results.

However, It's interesting that the first output of Ruby 3.1.2 and Ruby 3.1.2 + YJIT were rather low, then the numbers grew gradually. Rather than "warming up", it performed like "cooling down". Some extra efforts are needed to figure out why Ruby 3.1.2 will have this kind of behavior on SerpApi.

The user time of Ruby 3.1.2 and Ruby 3.1.2 + YJIT (the first column) were very close. However, the latter took much more time on sys time (the second column), resulting in a slower time overall.

We guess that the reason why SerpApi was not improved by YJIT is that Nokogiri, the HTML parser we are using, is mostly written in C extension. JIT compilers cannot optimize C code. And there may be extra overhead calling C code from ruby when JIT compiler is enabled.

We are also attaching memory consumption profiled by memory_profiler.

We've run twice for each ruby version because the first run seems to allocate much more memory than the rest. But the overall results were the same. Ruby 2.7.2 allocated the least memory, followed by Ruby 3.1.2 + YJIT, Ruby 3.1.2 consumed the most.

Conclusion

It can be concluded the new version of Ruby was not performing well on SerpApi. We are sensitive to performance, and it seems we might not migrate to Ruby 3 for now. But these are just early results of the whole benchmark schedule. We will also compare Rails 6 vs Rails 7 and other new versions of libraries to make the final decision.

Thanks for reading, see you in the next benchmark blog post.