Web Scraping

Tagged

A place for all web scraping tutorials in any language.

Scrape Related Searches from DuckDuckGo using Python

Scrape query name and a link to that query from DuckDuckGo related search results using Python.

Scrape DuckDuckGo Organic Results with Python

It's a good idea to focus not only on one search engine since DuckDuckGo users get a higher conversion rate and tend to have a lower bounce rate.

Google Shopping

Scrape Google Shopping Product Results with Python

This blog post is about scraping product title, description, specification, reviews, buying options, user reviews from Google Shopping.

Google

Scrape Google Local Place from Organic Search with Python

How to scrape the title, address, phone, hours, options, website, and directions links from Google Local Place Results using Python with beautifulsoup, requests, lxml libraries.

Web Scraping

How we reverse-engineered Google Maps pagination

In this story, you'll see the process of decoding URL parameters for pagination on Google Maps. It involved deobfuscation of Closure-compiled JavaScript, reverse-engineering of Protobuf data structures, and a bit of math. We tried to decode URL parameters by ourselves, by using pbtk, and attempted to outsource this

APIs

Easy Integrations

Features

Use Cases

Resources