How to Scrape Google News Results

Andy L -

Google News is a special news site that aggregates headlines from news sources worldwide.

Ran by algorithms, Google News contains real-time information. Google News covers a wide range of topics and is always updated

.

Scraping Google News results can provide you with valuable data and insights, allow you to access the latest news articles. More than that, you can scrape by topics, keywords or sources.

Analyzing scraped news data can give you ideas of trends, sentiments and events across various industries and regions.

You can also monitor your competitors covered in the news, gain more insights into market movements and make better decisions for your business.

Scraping Google News is challenging for some reasons: Anti-Scraping Measures, Dynamic website, HTML structure changes, Data volume, CAPTCHAs and challenges,...

Nowadays, every company prefers to use SERP API providers rather than building the scraper from scratch. SerpApi is one of the best SERP API providers which supports scraping

Google News Results without headache.


Setting up a SerpApi account

SerpApi offers a free plan for newly created accounts. Head to the sign-up page to register an account and complete your first search with our interactive playground. When you want to do more searches with us, please visit the pricing page.

Once you are familiar with all results, you can utilize SERP APIs using your API Key.

Scrape your first Google News results with SerpApi

Head to the Google News Results from the documentation on SerpApi for details.

In this tutorial, we will scrape and extract the organic results for all "iphone" news. The data contains: "position", "title", "link", "date", "source", "description", "thumbnail", and more. You can also scrape more information with SerpApi.

First, you need to install the SerpApi client library.

pip install google-search-results

Set up the SerpApi credentials and search.

from serpapi import GoogleSearch
import os, json

params = {
    'api_key': 'YOUR_API_KEY',         # your serpapi api
    'engine': 'google',               # SerpApi search engine	
    'tbm': 'nws',                     # Google News engine 
    'q': 'iphone'
}

To retrieve Google News Results for a given search query, you can use the following code:

results = GoogleSearch(params).get_dict()['news_results']

You can store Google News Results JSON data in databases or export them to a CSV file.

import csv

header = ['position', 'title', 'link', 'date', 'source', 'description', 'thumbnail']

with open('google_news.csv', 'w', encoding='UTF8', newline='') as f:
    writer = csv.writer(f)

    writer.writerow(header)

    for item in results:
        print(item)
        writer.writerow([item.get('position'), item.get('title'), item.get('link'), item.get('date'), item.get('source'), item.get('description'), item.get('thumbnail')])

This example is using Python, you can use your favorite programming languages likes Ruby, NodeJS, Java, PHP....

If you have any questions, please feel free to contact me.