A CPU benchmark of cloud providers: DigitalOcean vs. Vultr vs. Equinix

SerpApi is CPU-intensive. A search usually takes over 500ms to complete. We are constantly looking for ways to improve the search performance of the API.

Apart from optimizing the code, a simple and effective approach will be to upgrade the hardware. You easily get performance upgrades by paying more money. But for CPU-intensive servers, a more powerful CPU directly leads to lower response time, leading to fewer concurrent workers needed and, thus, fewer servers, potentially saving money.

We will now compare cloud providers' CPU benchmarks, including DigitalOcean, Vultr, and Equinix. We tested both virtual servers and bare metal servers.

Initially, we used sysbench to run CPU benchmarks. But we soon found that sysbench produces very unreliable results on CPU scores, which cannot reflect the performance of SerpApi. We later switched to 7z b -mmt1 and verified that its results were linearly correlated with SerpApi performance.

Here are the results.

The results were quite interesting to study.

For the virtual servers, Vultr was generally faster than DigitalOcean by up to 10% for shared CPUs and up to 33% for dedicated CPUs.

It was weird that servers in Los Angeles or San Francisco were more performant than servers in New York or New Jersey, both DigitalOcean and Vultr. We would have to benchmark all locations to pick a winner, but let's do that later.

Vultr AMD servers were generally slower than Intel ones. We later benchmarked DigitalOcean servers too, and it was the same result! Unfortunately, they sell them at the same price.

We got a very low CPU score on DigitalOcean New York Datacenter 3, both shared CPU and dedicated CPU. The score became normal the next day when we ran the benchmark the second time. The cause of the low score can be noisy neighbors, but unexpectedly, they can be 2x-3x slower. Dedicated CPU servers were also affected. Shouldn't they "have guaranteed access to the full hyperthread at all times" according to the documentation?

For bare metal servers, Equinix m3.small.x86 was the absolute winner. It's 22% faster than Vultr AMD EPYC 7443P. But the price of the Equinix one was higher too. The Vultr one was the cost-performance choice.

Bare metal servers are guaranteed to provide complete system resources for CPU-intensive tasks. We are willing to try them out in production.