Integrate SerpApi with Pipedream (No Code)

Pipedream is a powerful platform that allows you to automate workflows without writing code and integrate various services effortlessly. Pipedream’s low-code environment makes it easy to build, deploy, and build and scale custom workflows quickly without the need for managing infrastructure.

Scraping search engine results in Pipedream can allow you to get useful data for your use case - this can range from getting top 10 organic links from Google search results to scraping details of videos on Youtube. SerpApi offers APIs that you can use within Pipedream to scrape data from search results as part of any workflow.

Why Use SerpApi?

SerpApi manages the intricacies of scraping and returns structured JSON results, which allows you to save time and effort. We take care of proxies and any CAPTCHAs that might be encountered, so that you don't have to worry about your searches being blocked.

We also do all the work to maintain all of our parsers. This is important, as Google and other search engines are constantly experimenting with new layouts, new elements, and other changes. By taking care of this for you on our side, we eliminate a lot of time and complexity from your workflow.

How To Get Started

In this guide, I'll walk you through how to set up SerpApi with Pipedream, which will enable you to automatically gather and process data from any of the search engines we scrape after some basic setup. To get started, you need to set up a couple things -

1. SerpApi Account: You'll need an API key from SerpApi. If you don't already have an account, sign up for SerpApi. Once you complete the account verification process, you can access your SerpApi key on your dashboard.
2. Pipedream Account: Create a Pipedream account.

Set up your SerpApi API key in Pipedream

You can get your SerpApi key from the dashboard, or the Manage API Key page after logging into your SerpApi account.

Once you have that, head over to the Account's tab in Pipedream and add the key there as shown below:

After that, set your API key like this:

Make your first API request

We'll be using Pipedream's SerpApi integration to get data from SerpApi's APIs.

Start by creating a new workflow. From the Pipedream dashboard, click on the "New Workflow" button. This will bring up a new workflow creation window:

Set Up The Trigger Step

Choose whether you want your workflow to be triggered manually or automatically via a webhook, HTTP request, or any of the other available triggers.

For the sake of this tutorial, let’s use an HTTP trigger to start our workflow. You can adjust this later to use other triggers, like scheduling or event-driven actions.

Here is what that looks like:

Once you save this, you will need to generate a test event and verify it works before you are able to use this in your workflow:

Following this, you're ready to use this trigger in your workflow. We can move on to configuring the action step now.

Set Up The Action Step

Let’s make a request to SerpApi to scrape the search engine results. SerpApi offers a variety of search engines to scrape, including Google, Bing, and more. You can check the full list of our APIs on the left panel under the following link: https://serpapi.com/search-api

Now, let's add our action step in the Pipedream workflow. Select "SerpApi" from the actions dropdown:

These are the possible actions which will show up:

In this blog post, we'll explore the "Build any SerpApi API request" action and see how you can use this to query any of our APIs. This is what it initially looks like:

Next, add your API key to this action step:

Once that's done, you can add all the other parameters and test your query. And that's it, you have just set up your first request in Pipedream. Following this, you can deploy it once you have set it up for your use case.

Example Use Case: Get AI overviews from Google Search

For the purposes of this tutorial, let's consider setting up a Pipedream workflow to scrape AI overviews from Google Search responses.

When we query the Google Search API (engine = "google"), Google sometimes returns an AI Overview, which we include directly in the response, as seen here:

View it in Playground

However, Google may not always return the actual overview, but a page_token instead. When this happens, we have to send an additional request to the AI Overview API to retrieve the full AI Overview. We can do this using the page_token . The page_token is only valid for 4 minutes after the initial request.

Let's implement a Pipedream workflow to get the AI overview for any query.

After the initial setup we did in the previous sections, we can configure the first request with a sample query for which we expect an AI overview to show up. In this case, I'm using the query: "how does art work in android?"

Following this configuration, you can test it and should see the ai_overview block show up. If the response includes text_blocks, then it means you've received the AI overview directly in the response. It may instead include a page_token, which means we need to make a follow up request to the AI overviews API to get the AI overview for that search.

In this case, we can use an IF condition in Pipedream, since we only want to make a call to our AI overview API if we receive a page token in response from the Google Search API:

We need to check if the ai_overview field exists, and if the page_token exists within it. If so, we will make a call to the AI overviews API. This is going to be the IF condition check:

Then, we can configure the AI overview search with the google_ai_overview engine and the page_token from the previous search's response.

It's now complete! We can test this workflow with different queries, and see if it's working as expected.

Here's a demo test run:

Here's another one with the query "Coffee" where this IF condition should not match as the response does not contain an ai_overviews part. Thus, the workflow stops after the Google Search API response.

Conclusion

Using SerpApi with Pipedream is a powerful combination for automating search engine data collection. By following these steps, you’ve created a workflow that triggers SerpApi requests, processes the search results, and optionally can send the data to another service. 

I hope you found this tutorial helpful. If you have any questions, don't hesitate to reach out to me at sonika@serpapi.com.

Related Links and Posts

• SerpApi's Google Search Engine Results API
• SerpApi's Google AI overviews API
• SerpApi's Playground
• Pipedream Integration

Explore other ways to use SerpApi without writing any code:

No Code SERP API

Collect data from Search Engine results page like Google Search, Google Maps, and others without writing a single line of code.

Announcing SerpApi’s Make App

We are pleased to announce the release of our Custom App on Make.com. This makes it easier than ever to integrate SerpApi with other systems without having to write any code.

SerpApiAlex Barron

Connect SERP API with Google Sheets (No Code)

Here is how to connect SERP API (SerpApi) to Google Sheets using a no-code solution

SerpApiHilman Ramadhan

How to Scrape Google Search Results in Bubble.io using SerpApi

Learn about how to scrape Google search results in Bubble.io using SerpApi’s Google Search API without writing any code

SerpApiSonika Arora