Unlocking a Vast Range of Search Engines in Auto-GPT with Comprehensive Result Scraping

The landscape of web search has expanded exponentially, and extracting relevant information from search engine results pages (SERPs) is a crucial aspect of many AI applications. Auto-GPT, one of the most popular AI applications, has built-in web scrapers that return structured JSON results from Google or DuckDuckGo. With plugins, it can also get results from Bing and Baidu as of this writing. However, none of them can get rich data from SERPs, limiting Auto-GPT to providing more valuable insights.

In contrast, SerpApi excels at collecting an extensive range of data elements, including answer boxes, knowledge graphs, answers to frequently asked questions, and more. This comprehensive scraping capability can empower Auto-GPT to generate more accurate, detailed, and contextually rich responses to user queries.

In addition, SerpApi also offers a broader range of search engines such as Google, Yahoo, Yandex, Naver, and many more. This broader range allows developers to tap into diverse search platforms, ensuring comprehensive coverage and access to a wealth of data.

We've developed a plugin - Auto-GPT-SerpApi-Plugin - that integrates SerpApi into Auto-GPT. This plugin allows users to enable SerpApi as the default search engine in Auto-GPT. In this blog post, we'll show you how to use it to improve Auto-GPT's search results.

Installing SerpApi plugin

Before setting up the plugin, make sure you have a working Auto-GPT installation. Follow the official documentation for installation.

Next, download the plugin as a zip archive.

curl -L -o ./Auto-GPT-SerpApi-Plugin.zip https://github.com/SerpApi/Auto-GPT-SerpApi-Plugin/archive/refs/heads/master.zip

If you are running Auto-GPT in with docker, you need to add the zip file into the volumes section of docker-compose.yml

version: "3.9"
services:
  auto-gpt:
    image: significantgravitas/auto-gpt
    ...
    volumes:
      ...
      - /path/to/Auto-GPT-SerpApi-Plugin.zip:/app/plugins/Auto-GPT-SerpApi-Plugin.zip

Otherwise, simply put Auto-GPT-SerpApi-Plugin.zip in the plugins folder of Auto-GPT.

Using SerpApi plugin

The most straightforward setup to enable SerpApi plugin is to set SERPAPI_API_KEY as an environment variable in the Auto-GPT runtime.

To do that, open .env file within Auto-GPT, and append the following lines.

################################################################################
### SerpApi
################################################################################

SERPAPI_API_KEY=your_serpapi_key
SERPAPI_ENGINE=
SERPAPI_NO_CACHE=
SERPAPI_RESULT_FILTER=

At this stage, you only need to fill in SERPAPI_API_KEY, we'll cover the rest of the variables later.

To get a SerpApi API Key, sign up for a free account at SerpApi if you don't have one already. Navigate to the Dashboard page and locate "Your Private API Key".

Next, search for ALLOWLISTED_PLUGINS in the .env file and add this plugin.

################################################################################
### ALLOWLISTED PLUGINS
################################################################################

#ALLOWLISTED_PLUGINS - Sets the listed plugins that are allowed (Example: plugin1,plugin2,plugin3)
ALLOWLISTED_PLUGINS=AutoGPTSerpApiSearch

Now you have completed all the steps for a basic setup. Run Auto-GPT and make queries. Make sure you see SerpApi-Search-Plugin loaded in the welcome screen.

Internally, the google command will be intercepted and use the serpapi_search command instead. It's also possible that Auto-GPT uses the serpapi_search command directly.

You can always check the searches performed by Auto-GPT in Your Searches and the complete results for those searches.

Optimizing search results

SerpApi's JSON outputs can sometimes be too large for Auto-GPT/ChatGPT to handle effectively. To address this issue, we have implemented a configurable filter that allows users to select specific fields, reducing the overall content size. To customize the filter, you can utilize the SERPAPI_RESULT_FILTER variable, the usage of which is explained in the documentation.

By default, the filter value is set to organic_results(title,link,snippet), which means only the organic_results field is selected from the results. Within each organic result, only the title, link, and snippet are included. This default setting works well for most searches.

However, if you wish to provide Auto-GPT with more precise results and enable it to generate better outputs, you have the freedom to adjust the filter accordingly. For instance, answer_box, knowledge_graph, or related_questions fields often yield more straightforward answers to search queries than the organic_results field. It's important to note that these alternative fields may not always be available. Ideally, we aim for the most precise response while keeping the content size as compact as possible. For instance, answer_box is preferred over answer_box, organic_results. When you are certain that answer_box exists, it is recommended to use it alone.

Conclusion

By providing access to a broader range of search engines and scraping comprehensive data from result pages, SerpApi empowers developers to unlock the full potential of their AI applications. The integration of these powerful tools marks a significant step towards improving search-based AI applications and delivering more intuitive and intelligent user experiences. As the technology continues to evolve, we can expect even more exciting advancements in the field of web search and natural language processing.