Scrape Bing Answer Box with Python

Terry Tan -

Google has rich SERP features for sometime now, one of it is Answer Box. If you are interested in scraping Google Answer Box, feel free to checkout our Google Answer Box API. As for Bing, they have been making a lot of updates to their SERP features recently and we observe various Answer Box appear in their SERP. We are doing our best to support what available and recently we added the support for Bing Answer Box API. Mainly we have qna and fact answer box, we surely will expand to support more answer box type. The benefits of using our API is it allows you to scrape SERP results from Bing searches without any headache to manage document changes, bypass anti scraping protection, and proxies. In this article, I will demo how to retrieve the answer box data in only a few simple steps using Python.

Playground

You can try it out in our playground for the above search (no sign up required). This is the actual data from the search using our API.

"answer_box": {
  "type": "qna",
  "snippet_attributions": [
    {
      "text": "Here are the top 10 best NBA players of all time, according to various sources",
      "sources": [
        {
          "text": "1",
          "link": "<omitted for easier viewing...>"
        },
        {
          "text": "2",
          "link": "<omitted for easier viewing...>"
        }
      ]
    }
  ],
  "snippet": "Here are the top 10 best NBA players of all time, according to various sources:Michael Jordan (1985-2003)LeBron James (2004-Present)Kareem Abdul-Jabbar (1969-1989)Magic Johnson (1979-1996)Bill Russell (1956-1969)Wilt Chamberlain (1959-1973)Tim Duncan (1997-2016)Larry Bird (1979-1992)Kobe Bryant (1996-2016)Shaquille O'Neal (1992-2011)",
  "list": [
    "Michael Jordan (1985-2003)",
    "LeBron James (2004-Present)",
    "Kareem Abdul-Jabbar (1969-1989)",
    "Magic Johnson (1979-1996)",
    "Bill Russell (1956-1969)",
    "Wilt Chamberlain (1959-1973)",
    "Tim Duncan (1997-2016)",
    "Larry Bird (1979-1992)",
    "Kobe Bryant (1996-2016)",
    "Shaquille O'Neal (1992-2011)"
  ],
  "sources": [
    {
      "title": "Top 10 best NBA players of all time - Sportskeeda",
      "link": "<omitted for easier viewing...>",
      "snippet": "Top 10 best NBA players of all time1. Michael Jordan (1985-2003)2. LeBron James (2004-Present)3. Kareem Abdul-Jabbar (1969-1989)4. Magic Johnson (1979-1996)5. Bill Russell (1956-1969)More items",
      "highlighted_snippets": [
        "Top 10 best NBA players of all time"
      ],
      "displayed_link": "www.sportskeeda.com/basketball/top-10-greatest-…"
    },
    {
      "title": "The 100+ Best NBA Players Of All Time, Ranked by Fans",
      "link": "<omitted for easier viewing...>",
      "snippet": "If you've ever asked, \\"Who's the greatest basketball player of all time?\\", look no further. Ranked by votes from thousands of fans, this list features the all time NBA GOATs, including Michael Jordan, Kobe Bryant, LeBron James, Magic Johnson, Julius Erving, Wilt Chamberlain, and Larry Bird.",
      "highlighted_snippets": [
        "Michael Jordan, Kobe Bryant, LeBron James, Magic Johnson, Julius Erving, Wilt Chamberlain, and Larry Bird"
      ],
      "displayed_link": "www.ranker.com/crowdranked-list/the-top-nba-play…"
    }
  ],
  "images": [
    {
      "link": "<omitted for easier viewing...>",
      "thumbnail": "<https://www.bing.com/th?id=OIP.DuFkdq4LDZ3sttBF5zuZ6wHaD9&w=257&h=160&c=8&rs=1&qlt=90&o=6&pid=3.1&rm=2>"
    },
    {
      "link": "<omitted for easier viewing...>",
      "thumbnail": "<https://www.bing.com/th?id=OIP.oCnkx3EOaedg6xkSjWlbngHaEm&w=221&h=160&c=8&rs=1&qlt=90&o=6&pid=3.1&rm=2>"
    },
    {
      "link": "<omitted for easier viewing...>",
      "thumbnail": "<https://www.bing.com/th?id=OIP.P9nymW5Ap-BMq25XSM90RwHaEK&w=244&h=160&c=8&rs=1&qlt=90&o=6&pid=3.1&rm=2>"
    },
    {
      "link": "<omitted for easier viewing...>",
      "thumbnail": "<https://www.bing.com/th?id=OIP.R68t8X9zUZkoRnKVnEwJywHaFj&w=184&h=160&c=8&rs=1&qlt=90&o=6&pid=3.1&rm=2>"
    }
  ]
},

We try our best to cover as much data as possible as you can see the API return images sources, snippet attribution and many more.

The Code

Prerequisite:

Install SerpApi's Python library

pip install google-search-results

import os
from serpapi import BingSearch

SERPAPI_KEY = os.environ['SERPAPI_KEY']

search = BingSearch({
  "q": "top 10 nba players of all time",
  "location": "Austin,Texas",
  "api_key": SERPAPI_KEY
})

result = search.get_dict()

if "answer_box" in result:
  print(result["answer_box"])
else:
  print("No answer box")

This code is available in Replit, you can fork it and play around with it.

Breakdown of the code:

  1. from serpapi import BingSearch: This imports the "BingSearch" class from the "serpapi" library, which allows you to interact with the SerpApi service to retrieve search results.
  2. Set the SERPAPI_KEY : You can get a free API key upon register a new account in SerpApi. Once you register, head over to the Dashboard and copy the API key.
  3. Define the parameters: q is the query we want Bing to perform for the search. There are more parameters in our documentation, feel free to check it out.
  4. result = search.get_dict(): Initiates a search through our API, which triggers the retrieval of Bing Search Engine Results Pages (SERP) via proxy servers. Subsequently, the HTML content of the retrieved page is parsed and result is return in JSON format.
  5. Finally, we display the result if answer_box is present. This can be extend to various use cases.

Potential Use Cases

Scraping the Bing answer box can be useful for a variety of applications, such as:

  • Creating a chatbot that can answer users' questions
  • A voice assistant that can answer general questions
  • Conducting research on common questions and their answers
  • Monitoring changes in the answer box for a particular query or topic

Documentation

If you have any questions, please feel free to reach out to me.

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞