Ruby 3 has been released for a while. As a core language in SerpApi, the new version is definitely going to be tried out, especially given that Ruby 2.6 has already reached end-of-life, and it will soon come to Ruby 2.7, which SerpApi is currently running on. Apart from the new language features introduced in Ruby 3, performance is our main concern. Although there have been many comparisons between Ruby 3.1 and Ruby 2.7, we want to test them with our own environment to see if Ruby 3 will have a performance improvement on SerpApi.
Ruby 3.1 has shipped with a new JIT compiler, YJIT. It claims to have performance improvements on most real-world software, up to 22% on railsbench
and 39% on liquid-render
. We will be comparing Ruby 2.7.2, Ruby 3.1.2 and Ruby 3.1.2 + YJIT.
All the following benchmarks were run on a DigitalOcean CPU-Optimized, 2 vCPUs, 4 GB
virtual machine.
Benchmark on yjit-bench
Our first benchmark ran on yjit-bench, provided by Shopify, the developer of YJIT.
Run the Ruby 3.1.2 and Ruby 3.1.2 + YJIT benchmark by switching to Ruby 3.1.2 and running ./run_benchmarks.rb
Run the Ruby 2.7.2 benchmark by switching to Ruby 2.7.2 and running ./run_benchmarks.rb -e ruby
Here are the results. Lower is better.
Raw data:
benchmark | ruby2.7.2 | ruby3.1.2 | ruby3.1.2+yjit |
---|---|---|---|
30k_ifelse | 2835.369178 | 2871.740415 | 347.6600132 |
30k_methods | 7668.80311 | 7558.231579 | 825.0258768 |
activerecord | 194.1947169 | 192.832139 | 134.1704337 |
binarytrees | 344.994329 | 442.7235988 | 331.8990998 |
cfunc_itself | 87.15214192 | 98.89735636 | 45.6197738 |
chunky_png | 858.1087336 | 970.3283633 | 671.3995022 |
erubi | 452.75688 | 474.8846801 | 407.0801165 |
erubi_rails | 34.6515834 | 35.59898836 | 25.86827893 |
etanni | 525.0555173 | 604.5594306 | 843.4775177 |
fannkuchredux | 4544.754985 | 6215.255843 | 6271.386797 |
fib | 190.5258242 | 220.4583379 | 57.74113797 |
getivar | 105.105136 | 92.75804685 | 39.73911569 |
hexapdf | 3403.161764 | 3296.487006 | 2545.836672 |
keyword_args | 228.9924551 | 271.8941905 | 52.07178436 |
lee | 1078.889968 | 1145.596831 | 839.7007101 |
liquid-render | 181.0265133 | 205.334054 | 133.918901 |
168.2398866 | 181.6809455 | 167.7362736 | |
nbody | 118.3354334 | 110.2136637 | 83.34042853 |
optcarrot | 5182.830082 | 5723.337052 | 3678.027454 |
psych-load | 2598.436543 | 2434.784967 | 1932.677771 |
railsbench | 3825.04206 | 3505.656024 | 2687.687757 |
respond_to | 238.2646554 | 248.3746187 | 184.7939404 |
ruby-lsp | 105.0971961 | 130.3235934 | |
rubykon | 10704.44953 | 12166.72257 | 6502.561311 |
setivar | 67.84215721 | 69.57942243 | 47.23990744 |
str_concat | 121.3709915 | 131.0851263 | 130.3337132 |
The results showed that Ruby 3.1.2
is around the same performance as Ruby 2.7.2
, as reported by other benchmarks. YJIT turned out to outperform the other two a lot, which is quite impressive. railsbench
, that we paid the most attention to, was improved by 23.3% in this test. These results gave us the confidence to carry on to the next benchmark on SerpApi.
Benchmark on SerpApi
We benchmarked the most time-consuming process of SerpApi: HTML parsing and JSON output construction. A static google search result page was fed for the tests. Both time and memory usage were recorded. The following script was used for benchmarking.
def do_parse
Search.new(args_of_a_google_result_page).parse!
end
10.times {
puts Benchmark.measure {
30.times { do_parse }
}
}
require 'memory_profiler'
MemoryProfiler.report { do_parse }.pretty_print
Here are the results of the time elapsed. Lower is better.
Raw data:
Ruby 2.7.2
---------------
25.047808 0.161161 25.208969 ( 25.268428)
23.927413 0.035128 23.962541 ( 23.980549)
24.188614 0.031934 24.220548 ( 24.257346)
28.051661 0.003824 28.055485 ( 28.117897)
25.007042 0.039772 25.046814 ( 25.066986)
24.256959 0.023895 24.280854 ( 24.298495)
23.700391 0.000000 23.700391 ( 23.718018)
24.739665 0.019597 24.759262 ( 24.777065)
23.896913 0.024010 23.920923 ( 23.941751)
24.410275 0.019975 24.430250 ( 24.458073)
Ruby 3.1.2
---------------
23.831084 0.142897 23.973981 ( 24.013818)
25.364707 0.035620 25.400327 ( 25.435810)
26.620177 0.020064 26.640241 ( 26.657792)
27.575943 0.024074 27.600017 ( 27.619430)
27.609700 0.015994 27.625694 ( 27.645148)
27.890730 0.011946 27.902676 ( 27.921206)
28.092639 0.015964 28.108603 ( 28.127349)
27.807529 0.035993 27.843522 ( 27.862086)
27.865685 0.047982 27.913667 ( 27.933122)
27.976437 0.031967 28.008404 ( 28.027206)
Ruby 3.1.2 + YJIT
---------------
23.583271 1.066498 24.649769 ( 24.656726)
25.698567 0.884720 26.583287 ( 26.593125)
27.064727 0.955691 28.020418 ( 28.039999)
27.833003 0.923952 28.756955 ( 28.777363)
27.415354 0.963869 28.379223 ( 28.395398)
28.977574 0.945236 29.922810 ( 29.962006)
27.541010 0.947636 28.488646 ( 28.508437)
28.373851 0.899978 29.273829 ( 29.293450)
27.706739 0.983724 28.690463 ( 28.705540)
28.299704 0.907967 29.207671 ( 29.227521)
YJIT, the absolute winner of the previous benchmark, was not performing as expected, even worse than disabled. There are frustrating results.
However, It's interesting that the first output of Ruby 3.1.2
and Ruby 3.1.2 + YJIT
were rather low, then the numbers grew gradually. Rather than "warming up", it performed like "cooling down". Some extra efforts are needed to figure out why Ruby 3.1.2
will have this kind of behavior on SerpApi.
The user time
of Ruby 3.1.2
and Ruby 3.1.2 + YJIT
(the first column) were very close. However, the latter took much more time on sys time
(the second column), resulting in a slower time overall.
We guess that the reason why SerpApi was not improved by YJIT is that Nokogiri
, the HTML parser we are using, is mostly written in C extension. JIT compilers cannot optimize C code. And there may be extra overhead calling C code from ruby when JIT compiler is enabled.
We are also attaching memory consumption profiled by memory_profiler.
We've run twice for each ruby version because the first run seems to allocate much more memory than the rest. But the overall results were the same. Ruby 2.7.2
allocated the least memory, followed by Ruby 3.1.2 + YJIT
, Ruby 3.1.2
consumed the most.
Conclusion
It can be concluded the new version of Ruby was not performing well on SerpApi. We are sensitive to performance, and it seems we might not migrate to Ruby 3 for now. But these are just early results of the whole benchmark schedule. We will also compare Rails 6 vs Rails 7 and other new versions of libraries to make the final decision.
Thanks for reading, see you in the next benchmark blog post.