Yicheng Zhou - SerpApi

Capturing HTTPS traffic on Android 7+ devices

Before Android 7, all apps automatically trusted user-installed certificates by default, making the inspection of HTTPS traffic relatively straightforward. One could simply install a self-signed certificate into the system, and apps would automatically trust that certificate. Subsequently, an MITM (man-in-the-middle) proxy, such as Charles or Fiddler, could be initiated to

reverse-engineering

Extracting information from Google's protobuf-encoded JSON response

Google's protobuf-encoded JSON response has long been a mystery for many developers. When we make API calls to Google services, we often receive deeply nested arrays in JSON responses, lacking discernible keys to interpret the values. This blog post delves into the challenges I faced while working with

Ruby

The Internals of Ruby Thread

Most major programming languages implement threads to run code in parallel. This is also true for Ruby. However, Ruby's threading model differs from languages like C# and Java - it is not fully multithreaded. Specifically, we are referring to MRI Ruby (also known as CRuby) in this blog

Auto-GPT Plugin

Unlocking a Vast Range of Search Engines in Auto-GPT with Comprehensive Result Scraping

The landscape of web search has expanded exponentially, and extracting relevant information from search engine results pages (SERPs) is a crucial aspect of many AI applications. Auto-GPT, one of the most popular AI applications, has built-in web scrapers that return structured JSON results from Google or DuckDuckGo. With plugins, it

Web Scraping

How to get Bing Shopping Search Results in JSON

We've recently launched a new API for scraping shopping search results at SerpApi - Bing Shopping API. This API is able to get structured search results of product data from Bing Shopping. This blog post will walk through its usage. Introduction The most straightforward way to try out

HTML Parser

Nokolexbor - a performance-focused HTML parser for Ruby

There aren't many choices of HTML parsers in the ruby ecosystem. The most obvious one would be Nokogiri, which we used heavily at SerpApi. As time passed, we became gradually unsatisfied with Nokogiri's performance. Though it's mainly relying on libxml2 which is an XML

Ruby

How to detect memory leak in Ruby C extension

Memory leak has been one of the most difficult problems to track down in the C/C++ world. It's also true when it comes to Ruby C extensions. We don't expect memory bloat for a long-running Ruby service. Fortunately, the community has developed great tools to

debugging

How to create Ruby C extension and debug in VS Code

When developing Nokolexbor [https://github.com/serpapi/nokolexbor], I found debugging with gdb or lldb was troublesome because I have to do everything with commands. For me, it will be more delightful to debug with an interactive GUI, which makes it convenient to control breakpoints and steps, inspect values and

A CPU benchmark of cloud providers: DigitalOcean vs. Vultr vs. Equinix

SerpApi is CPU-intensive. A search usually takes over 500ms to complete. We are constantly looking for ways to improve the search performance of the API. Apart from optimizing the code, a simple and effective approach will be to upgrade the hardware. You easily get performance upgrades by paying more money.

APIs

Easy Integrations

Features

Use Cases