How to report a bug on the SerpApi Public Roadmap to ensure it gets fixed quickly

Illia Zub -

Search engines like Google are dynamic. They are always changing, updating, and improving. One of the consequences of this operation is that scrapers need to be updated and maintained to be sure that they can always handle the most current HTML structure of the results pages.

Unfortunately for us, Google doesn’t give us, or anyone else who scrapes their results pages, a heads up about these changes and how best to plan for them. This means one of the primary ways we learn of an HTML change is when our users report missing JSON elements.

We didn’t always have a public roadmap for SerpApi but our primary reason for integrating one is so that our users can more quickly and efficiently report bugs to our engineering team directly. This being the case, it’s important to be sure to include helpful and necessary information for our engineering team to be able to reproduce, diagnose and patch the issue.

We’ve created many of the bug posts on our public roadmap on behalf of customers and here are some things I do to make sure the bug I’m reporting is clear and an engineer can get to patching it right away!

Examples

Including an example search of yours is the best way to directly show SerpApi engineers what happened with your specific search. The best way you can share your example is with a Search ID. Every search result JSON will have an assigned Search ID found under search_metadata. Here’s an example:

"search_metadata": {
  "id": "601d8c728f13dadfc5b57e18",
  "status": "Success",
  "json_endpoint": "https://serpapi.com/searches/6eccd428d7da60f9/601d8c728f13dadfc5b57e18.json",
  "created_at": "2021-02-05 18:20:34 UTC",
  "processed_at": "2021-02-05 18:20:34 UTC",
  "google_url": "https://www.google.com/search?q=Coffee&oq=Coffee&uule=w+CAIQICIaQXVzdGluLFRleGFzLFVuaXRlZCBTdGF0ZXM&hl=en&gl=us&sourceid=chrome&ie=UTF-8",
  "raw_html_file": "https://serpapi.com/searches/6eccd428d7da60f9/601d8c728f13dadfc5b57e18.html",
  "total_time_taken": 1.38
}

You can see the id is 601d8c728f13dadfc5b57e18. This is the search ID for this particular search.

Sometimes you don’t immediately realize that something went wrong with a search and it can be days before you even notice. Then, after you report the bug, more time can pass before an engineer sees it. During this time, the results on the page can change vastly so even reproducing the same search at a later date might not help to solve the problem.

That’s why it’s necessary to have a way to share a specific search from a specific moment. The Search ID is how you can do that. The search’s unique search ID is tied to that very specific instance and is tied to both the results JSON and the corresponding HTML page for that search. Both of these files are crucial for our engineers to be able to quickly patch a bug.

You can also use the search ID to share a search inspect link. Here’s an example of a search inspect link:

https://serpapi.com/search/601d8c728f13dadfc5b57e18/inspect

Inspect links are only viewable by the user who made the search and by SerpApi admins so you don’t have to worry about leaking anything specific to your account by posting search IDs or inspect links on the public roadmap.

Pictures

A picture is worth a thousand words and that’s just as true if not more when posting a bug on our public roadmap. Sharing a picture of the issue is an efficient way to ensure everyone is on the same page.

One thing you can do is be sure to highlight specific page elements that are relevant to the issue. There are lots of different ways to refer to the same thing so highlighting it in an image prevents any confusion. For example, if you had an issue where the “People also ask” element is missing from the JSON, you can include an image like this:

In this image, it’s clear what element on the page is in question and what our engineers should focus on in the example you shared.

Title

The title is how you communicate the core issue you’re experiencing and what needs to be fixed by the SerpApi engineering team. It also is how other public roadmap users find both open and closed bugs so it’s important that your post has a useful title.

The first thing I include when titling a bug post is the search engine where the bug happened. SerpApi supports the scraping of a wide variety of search engines and it’s common for elements to share the same names between engines. For example, both our Google Search API and our Bing Search API will have organic_results so it’s necessary to distinguish the engine to prevent this type of confusion.

Next, I state what didn’t work correctly. For example, if my search had organic_results but in the JSON they were all missing the title, I’d say something like:

`organic_results` are missing the `title` in the results JSON

In this example, it’s clear what the issue is about (organic_results missing the title) and where the issue is found (in the results JSON). By being specific, you decrease the chances an engineer has to personally follow-up on the thread just to understand what is actually going wrong. This only increases the time before an engineer can actually start working on the issue.

If we put both of these steps together, we’ll get something like this:

Google Search `organic_results` are missing the `title` in the results JSON

When an engineer reads this, they know the issue happened with Google search, where the organic_results were missing the title, in the results JSON.

Another thing to note would be the inclusion of the library in your integration is written in. Most of the API load is handled on SerpApi’s end and your integration mostly just makes and receives the request. Because of this, it’s very rare that we have issues specifically tied to a certain library or programming language. But if you do think your issue is specific to the library you’re using, be sure to include that in the title as well.

Description

The description is the place where you’ll include your search ID or inspect link examples in addition to any extra information that may be helpful in describing what went wrong. This can be helpful if there was a specific order of events for the issue or if the issue happens somewhat inconsistently. The description can also be a good place to include snippets of the results JSON or HTML to further demonstrate the issue.

Thanks for your help

We at SerpApi know that it isn’t fun to experience a bug let alone have to report it. Our API covers a vast amount of data across multiple platforms so there’s a lot to cover and maintain.

We’re a small team but we’re passionate about building a good, reliable API for our users and properly supporting everyone’s use cases. We always take bugs seriously and work to get them fixed as soon as possible. We really appreciate it when you take the time to post a bug you’ve experienced on our public roadmap and help us to be sure our API is working as it should.