This week we'll showcase how we benchmark our ML implementation we developed for SerpApi's Google Local Pack Scraper API on Rails.

Low Accuracy Benchmark for Overall Parsing Process (100 times)

We will use the same queries we used in last week's blog post to create a benchmark to report on milliseconds. However, one thing to mention here is that the average time taken in 100 times the same process will be subjected to a margin of error.

I will put another test on a singular result to measure more realistic numbers. This process will reflect how it will be effecting the code of the user who's making a search. The cases with odd behavior should be reflected regardless of the error margin.

Here's the benchmark code for Rails:

traditional_benchmarks = []
ml_benchmarks = []

100.times do
  traditional_benchmarks << Benchmark.ms { search.parse! false }
  ml_benchmarks << Benchmark.ms { search.parse! true }
end

traditional_average = traditional_benchmarks.sum / traditional_benchmarks.size
ml_average = ml_benchmarks.sum / ml_benchmarks.size
average_difference = ml_average - traditional_average 

if traditional_average < ml_average
  ratio_change = ml_average / traditional_average
  change = "slower"
else
  ratio_change = traditional_average / ml_average
  change = "faster"
end

puts "Traditional Parser Benchmark Average: #{traditional_average} ms"
puts "ML Hybrid Parser Benchmark Average: #{ml_average} ms"
puts "Average Difference: #{average_difference} ms"
puts "ML Hybrid Parser is #{ratio_change} times #{change} than Traditional Parser"

Dentist

 "local_results": {
    "places": [
      {
        "position": 1,
        "title": "Austin Cosmetic Dentistry by Tejas Patel, DDS",
        "address": "221 W 6th St #940",
        "rating": 4.9,
        "reviews": 275,
        "hours": "Closes soon ⋅ 4PM",
        "phone": "(512) 476-2336",
        "lsig": "AB86z5V7-Co76bptvuPt00IjA1ut",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5V7-Co76bptvuPt00IjA1ut&ludocid=17726167689886688143&q=Dentist&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.26795,
          "longitude": -97.74515
        }
      },
      {
        "position": 2,
        "title": "East Austin Dental",
        "address": "1000 E 5th St STE 110 · In Corazon Apartments",
        "rating": 4.8,
        "hours": "Closes soon ⋅ 4PM",
        "phone": "(512) 499-0067",
        "lsig": "AB86z5WbLP6dZDWYnq91fO356Est",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5WbLP6dZDWYnq91fO356Est&ludocid=16796499427254916480&q=Dentist&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.264153,
          "longitude": -97.73239
        }
      },
      {
        "position": 3,
        "title": "Austin Comestic Dentistry",
        "phone": "(512) 551-1880",
        "lsig": "AB86z5Wcc9TbOpkoDmgKyi1TUoUY",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5Wcc9TbOpkoDmgKyi1TUoUY&ludocid=1429802282623203746&q=Dentist&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.272924,
          "longitude": -97.7444
        }
      }
    ]
  },
-

Query: Dentist
Traditional Parser Benchmark Average: 344.3701663099887 ms
ML Hybrid Parser Benchmark Average: 349.1813165999997 ms
Average Difference: +4.811150290011028 ms
ML Hybrid Parser is 1.0139708684453235 times slower than Traditional Parser

Bookstore

  "local_results": {
    "places": [
      {
        "position": 1,
        "title": "BookPeople",
        "type": "Book store",
        "address": "603 N Lamar Blvd",
        "rating": 4.7,
        "delivery": "In-store shopping·Curbside pickup·Delivery",
        "hours": "Open ⋅ Closes 7PM",
        "phone": "(512) 472-5050",
        "lsig": "AB86z5VOIjaQM_OHJDDP78c75lF9",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5VOIjaQM_OHJDDP78c75lF9&ludocid=15954964542614962629&q=Bookstore&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.271843,
          "longitude": -97.752945
        }
      },
      {
        "position": 2,
        "title": "South Congress Books",
        "type": "Book store",
        "address": "1608 S Congress Ave",
        "rating": 4.5,
        "reviews": 203,
        "delivery": "In-store shopping·Curbside pickup",
        "hours": "Open ⋅ Closes 6PM",
        "phone": "(512) 916-8882",
        "lsig": "AB86z5WuflRhwRxW1_l4kcQe8HiA",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5WuflRhwRxW1_l4kcQe8HiA&ludocid=4735403951136453416&q=Bookstore&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.247662,
          "longitude": -97.750626
        }
      },
      {
        "position": 3,
        "title": "12th Street Books",
        "type": "Used book store",
        "address": "827 W 12th St",
        "rating": 4.3,
        "reviews": 13,
        "hours": "Temporarily closed",
        "phone": "(512) 499-8828",
        "lsig": "AB86z5WZ7ooDkCDty0PCVl-RnDqT",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5WZ7ooDkCDty0PCVl-RnDqT&ludocid=16276231750099540045&q=Bookstore&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.276222,
          "longitude": -97.74959
        }
      }
    ]
  },

Query: Bookstore
Traditional Parser Benchmark Average: 282.3834452000028 ms
ML Hybrid Parser Benchmark Average: 276.60691514000746 ms
Average Difference: -5.776530059995366 ms
ML Hybrid Parser is 1.0208835345171017 times faster than Traditional Parser

Mexican Restaurant

  "local_results": {
    "places": [
      {
        "position": 1,
        "title": "Tres Potrillos",
        "type": "Mexican",
        "delivery": "Dine-in·Takeout·No delivery",
        "lsig": "AB86z5UnM7a_iraiJwHbX4RTV74b",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5UnM7a_iraiJwHbX4RTV74b&ludocid=4949025990990656275&q=Mexican+Restaurant&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.267153,
          "longitude": -97.74306
        }
      },
      {
        "position": 2,
        "title": "Comedor",
        "type": "\"Their interpretation of mexican cuisine is exceptional.\"",
        "address": "501 Colorado St",
        "rating": 4.4,
        "reviews": 491,
        "hours": "Closed ⋅ Opens 5PM",
        "lsig": "AB86z5XVNkH5NYsRCN5t2FCdM89-",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5XVNkH5NYsRCN5t2FCdM89-&ludocid=6958240678106455106&q=Mexican+Restaurant&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.2676,
          "longitude": -97.74416
        }
      },
      {
        "position": 3,
        "title": "Iron Cactus Mexican Restaurant and Margarita Bar",
        "type": "Mexican",
        "price": "$$",
        "address": "606 Trinity St",
        "rating": 4.1,
        "delivery": "Dine-in·Takeout·No delivery",
        "lsig": "AB86z5X7Qq4vf--TBLEsXDYRUbVd",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5X7Qq4vf--TBLEsXDYRUbVd&ludocid=2855033105612448970&q=Mexican+Restaurant&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.2674,
          "longitude": -97.73945
        }
      }
    ]
  },

Query: Mexican Restaurant
Traditional Parser Benchmark Average: 225.45420559999457 ms
ML Hybrid Parser Benchmark Average: 223.7522506300229 ms
Average Difference: -1.7019549699716663 ms
ML Hybrid Parser is 1.007606426148472 times faster than Traditional Parser

Insurance Agency

  "local_results": {
    "places": [
      {
        "position": 1,
        "title": "Amco Insurance Austin",
        "type": "Insurance agency",
        "address": "RdB, 12603 Market Street",
        "rating": 1,
        "reviews": 1,
        "phone": "(713) 450-4949",
        "lsig": "AB86z5XbiSodHxmtytu2NF12jBla",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5XbiSodHxmtytu2NF12jBla&ludocid=3762069798187571614&q=Insurance+Agency&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.267153,
          "longitude": -97.74306
        }
      },
      {
        "position": 2,
        "title": "Michaels Insurance Group",
        "type": "Insurance agency",
        "address": "501 Congress Ave Suite 150A",
        "hours": "Open ⋅ Closes 6PM",
        "phone": "(512) 593-1171",
        "lsig": "AB86z5UOT_Ai1eBIwgwkGAIAMAF8",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5UOT_Ai1eBIwgwkGAIAMAF8&ludocid=16496161963606301439&q=Insurance+Agency&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.267204,
          "longitude": -97.74261
        }
      },
      {
        "position": 3,
        "title": "Jones Derek",
        "type": "Insurance agency",
        "address": "401 Congress Ave # 1900 · In Frost Bank Tower",
        "phone": "(512) 469-3430",
        "years_in_business": "10+ years in business",
        "lsig": "AB86z5VtQJgNsgG5T_iWQdrzIjPr",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5VtQJgNsgG5T_iWQdrzIjPr&ludocid=1165792112025048434&q=Insurance+Agency&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.266655,
          "longitude": -97.74313
        }
      }
    ]
  },

Query: Insurance Agency
Traditional Parser Benchmark Average: 243.73682075000943 ms
ML Hybrid Parser Benchmark Average: 242.52781047999179 ms
Average Difference: -1.209010270017643 ms
ML Hybrid Parser is 1.0049850376648553 times faster than Traditional Parser

Coffee

  "local_results": {
    "places": [
      {
        "position": 1,
        "title": "Houndstooth Coffee",
        "type": "Coffee shop",
        "price": "$$",
        "address": "401 Congress Ave #100c · In Frost Bank Tower",
        "rating": 4.6,
        "delivery": "Dine-in·Takeout·No delivery",
        "lsig": "AB86z5Vdw6C2pJpM0xQ6JUx2KONU",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5Vdw6C2pJpM0xQ6JUx2KONU&ludocid=11265938073076301333&q=Coffee&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.2664,
          "longitude": -97.74278
        }
      },
      {
        "position": 2,
        "title": "Starbucks",
        "type": "Coffee shop",
        "price": "$$",
        "address": "600 Congress Ave",
        "rating": 4.1,
        "reviews": 513,
        "delivery": "Dine-in·Drive-through·Delivery",
        "lsig": "AB86z5XTJ_Io_anVBu2fU6Zaqu3b",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5XTJ_Io_anVBu2fU6Zaqu3b&ludocid=10605736027611436825&q=Coffee&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.26826,
          "longitude": -97.74296
        }
      },
      {
        "position": 3,
        "title": "The Hideout Coffee House",
        "type": "Coffee shop",
        "price": "$",
        "address": "617 Congress Ave",
        "rating": 4.4,
        "delivery": "Dine-in·Takeout·No-contact delivery",
        "hours": "Closed ⋅ Opens 7AM Thu",
        "lsig": "AB86z5WSxdnDKVF_iLXNN6Lg0UQ5",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5WSxdnDKVF_iLXNN6Lg0UQ5&ludocid=15498522356495312950&q=Coffee&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.268572,
          "longitude": -97.742165
        }
      }
    ]
  },

Query: Coffee
Traditional Parser Benchmark Average: 381.3813184999935 ms
ML Hybrid Parser Benchmark Average: 374.56337533000345 ms
Average Difference: -6.817943169990031 ms
ML Hybrid Parser is 1.0182023754030494 times faster than Traditional Parser

Accurate Benchmark for Overall Parsing Process



The above benchmarks show how the implementation behaves in short term and it is showing no sign of odd cases.

However, let's implement the same test for a singular result with more accuracy. We'll run both parsing processes for 60 seconds and compare the run counts:

require "benchmark/ips"

Benchmark.ips do |x|
  x.config(:time => 60, :warmup => 2)
  x.report("Traditional Parser") { search.parse! false }
  x.report("ML Hybrid Parser") { search.parse! true }
  x.compare!
end

Here's the result:

  "local_results": {
    "places": [
      {
        "position": 1,
        "title": "Austin MacWorks",
        "type": "Computer store",
        "address": "450 W 2nd St · In 2nd Street District",
        "rating": 3.6,
        "reviews": 35,
        "delivery": "In-store shopping·In-store pickup",
        "hours": "Closes soon ⋅ 6PM",
        "phone": "(512) 476-7000",
        "lsig": "AB86z5UwIX4i7B78tBctQlVN-S5I",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5UwIX4i7B78tBctQlVN-S5I&ludocid=7461317996150463296&q=Computer+Store&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.265661,
          "longitude": -97.74838
        }
      },
      {
        "position": 2,
        "title": "Mactell Corporation",
        "type": "Computer store",
        "phone": "(512) 457-1133",
        "lsig": "AB86z5UVY5r6ZpGnUqP-wixtE-Nu",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5UVY5r6ZpGnUqP-wixtE-Nu&ludocid=1747786920379540476&q=Computer+Store&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.272924,
          "longitude": -97.7444
        }
      },
      {
        "position": 3,
        "title": "Campus Computer Store",
        "type": "Computer store",
        "address": "2304 Whitis Avenue, (FAC) 109 · In the University of Texas at Austin",
        "rating": 3.8,
        "reviews": 53,
        "delivery": "In-store shopping·Curbside pickup·Delivery",
        "hours": "Closes soon ⋅ 6PM",
        "phone": "(512) 475-6550",
        "lsig": "AB86z5XQ-nWlsJPMAzOoIwJ_Pjll",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5XQ-nWlsJPMAzOoIwJ_Pjll&ludocid=15624962612924024738&q=Computer+Store&tbm=lcl",
        "gps_coordinates": {
          "latitude": 30.286259,
          "longitude": -97.74032
        }
      }
    ]
  },
Calculating -------------------------------------
  Traditional Parser      5.741  (± 0.0%) i/s -    343.000  in  60.020695s
    ML Hybrid Parser      4.807  (± 0.0%) i/s -    288.000  in  60.158318s

Comparison:
  Traditional Parser:        5.7 i/s
    ML Hybrid Parser:        4.8 i/s - 1.19x  (± 0.00) slower

Benchmark for Local Parsing Process

Let's also define a local benchmark for the local parsing process. This is a direct comparison between the two methods. Here's the test for it:

require "benchmark/ips"

Benchmark.ips do |x|
  x.config(:time => 60, :warmup => 2)
  x.report("ML Hybrid Parser") { get_local_results_desktop_ml local_results_top }
  x.report("Traditional Parser") { get_local_results_desktop }
  x.compare!
end

Output:

Calculating -------------------------------------
  Traditional Parser     38.829  (± 2.6%) i/s -      2.328k in  60.015623s
    ML Hybrid Parser     23.121  (± 4.3%) i/s -      1.384k in  60.003592s

Comparison:
  Traditional Parser:       38.8 i/s
    ML Hybrid Parser:       23.1 i/s - 1.68x  (± 0.00) slower


Conclusion

If we take into account the accurate results, 1.68x slower local change and 1.19x slower for the entire parsing process (intrinsic to the search query, and only in results with local part), is still a nice translation in terms of milliseconds.

Considering the traditional parser is giving too many false results, this implementation will serve critical importance without causing unexpected delays in load time. I have seen it perform better in other searches.

The minimum I can get is 1.07x slower inaccurate results for the entire parsing process.

The maximum I have encountered is this result, thus I am sharing it as an example for honest representation.

In any case, it will be negligible to the user. I would like to thank the reader for the attention, and to the brilliant people of SerpApi for all their support. Next week, we'll talk about the general testing purposes of ML.