> ## Content Index
> Fetch the complete content index at: https://serpapi.com/blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# Web Scraping Google Arts & Culture Artist Results with Nodejs
- URL: https://serpapi.com/blog/web-scraping-google-arts-culture-artist-results-with-nodejs/
- Published: 2022-11-22T18:59:06.000Z
- Updated: 2022-12-07T13:38:22.000Z
- Description: A step-by-step tutorial on creating a Google Art & Culture Artists Results web scraper in Nodejs.
- Author: Mikhail Zub
- Tags: Google Arts & Culture, NodeJS, Web Scraping

## Intro

We currently don't have an API for Google Arts & Culture Artists page.

This blog post is written to show the DIY solution to extract data from "All, A-Z, Time" tabs that can be used for personal use while we're working on releasing our proper API.

The solution can be used for personal use as it doesn't include the [Legal US Shield](https://serpapi.com/#features) that we offer for our paid [production and above plans](https://serpapi.com/pricing) and has its limitations such as the need to bypass blocks, for example, CAPTCHA.

You can check our public roadmap to track the progress for this API:

🗺️

[\[New API\] Google Arts & Culture - Artists](https://github.com/serpapi/public-roadmap/issues/307)

## What will be scraped

![what](https://user-images.githubusercontent.com/64033139/199196096-76408a55-addc-4ee2-a853-97e5ce41ea7e.png)

## Full code

If you don't need an explanation, have a look at [the full code example in the online IDE](https://replit.com/@MikhailZub/Google-Arts-and-Culture-Artists-with-NodeJS-SerpApi#index.js)

```javascript
const axios = require("axios");

const AXIOS_OPTIONS = {
  baseURL: "https://artsandculture.google.com",
  headers: {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36",
  }, // adding the User-Agent header as one way to prevent the request from being blocked
  params: {
    hl: "en", // parameter defines the language to use for the Google search
  },
};
function getResultsFromCategory(categoryContent) {
  const artistsPattern = /cobject","(?<artist>[^"]+)","(?<works>[^ ]+) \w+","(?<thumbnail>[^"]+)","(?<link>[^"]+)/gm; //https://regex101.com/r/DXd78Z/1

  return [...categoryContent.matchAll(artistsPattern)].map(({ groups }) => ({
    artist: groups.artist,
    works: groups.works,
    thumbnail: `https:${groups.thumbnail}`,
    link: `${AXIOS_OPTIONS.baseURL}${JSON.parse(`"${groups.link}"`)}`,
  }));
}

function getArtistsInfo() {
  return axios.get("/category/artist", AXIOS_OPTIONS).then(function ({ data }) {
    const results = {};

    const popularCategoryPattern = /"PopularAssets:(?<content>.+?)\["stella\.pr/gm; //https://regex101.com/r/kyTsgs/1
    [...data.matchAll(popularCategoryPattern)].forEach(({ groups }) => (results.popular = getResultsFromCategory(groups.content)));

    const azCategoryPattern = /"(?<letter>[^"])",\["stella\.pr","(?<content>.+?)[\w"||\d]\]{2,3},\[/gm; //https://regex101.com/r/u1KZFf/1
    [...data.matchAll(azCategoryPattern)].map(({ groups }) => (results[groups.letter] = getResultsFromCategory(groups.content)));

    const timeCategoryPattern = /\[{1,2}"(?<time>[^"]{3,8})","?\w{4,7}.+?\["stella\.pr","DatedAssets(?<content>.+?)"?\d{3,5}"\]/gm; //https://regex101.com/r/5JHaQB/1
    [...data.matchAll(timeCategoryPattern)].map(({ groups }) => (results[groups.time] = getResultsFromCategory(groups.content)));

    return results;
  });
}

getArtistsInfo().then((result) => console.dir(result, { depth: null }));

```

## Preparation

First, we need to create a Node.js\* project and add [npm](https://www.npmjs.com/) package [axios](https://www.npmjs.com/package/axios) to make a request to a website.

To do this, in the directory with our project, open the command line and enter:

```bash
$ npm init -y   # create new project

```

And then:

```bash
$ npm i axios  # add axios package

```

\*If you don't have Node.js installed, you can [download it from nodejs.org](https://nodejs.org/en/) and follow the installation [documentation](https://nodejs.dev/learn/introduction-to-nodejs).

## Process

We need to extract artists data from the `script` tags using different [regular expressions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular%5FExpressions). The gif below shows you how the data looks like in the [page source](https://www.computerhope.com/issues/ch000746.htm):

![how](https://user-images.githubusercontent.com/64033139/199201182-ab3de23d-fcf0-49f8-8faa-e5e60acbcc7c.gif)

### Code explanation

First, we need to declare constant from [axios](https://www.npmjs.com/package/axios) library and write a request options: [HTTP headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) with [User-Agent](https://developer.mozilla.org/en-US/docs/Glossary/User%5Fagent) which is used to act as a "real" user visit, and the necessary parameters for making a request.

[Default axios request user-agent is axios/<axios\_version>](https://github.com/axios/axios/blob/892c241773e7dda78a969ac1faa9b365e24f6cc8/lib/adapters/http.js#L224) so websites understand that it's a script that sends a request and might block it. [Check what's your user-agent](https://www.whatismybrowser.com/detect/what-is-my-user-agent/):

```javascript
const axios = require("axios");

const AXIOS_OPTIONS = {
  baseURL: "https://artsandculture.google.com",
  headers: {
    "User-Agent": 
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36",
  }, // adding the User-Agent header as one way to prevent the request from being blocked
  params: {
    hl: "en", // parameter defines the language to use for the Google search
  },
};

```

Next, we write a function that extracts and returns artists data:

```javascript
function getResultsFromCategory(categoryContent) {
    ...
}

```

In this function we need to declare RegEx [artistsPattern](https://regex101.com/r/DXd78Z/1), then using [spread syntax](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread%5Fsyntax) we make and return an array ([map()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global%5FObjects/Array/map) method) from an [iterable iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators%5Fand%5FGenerators) of matches, received from [matchAll](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global%5FObjects/String/matchAll) method.

```javascript
    //https://regex101.com/r/DXd78Z/1
  const artistsPattern = 
        /cobject","(?<artist>[^"]+)","(?<works>[^ ]+) \w+","(?<thumbnail>[^"]+)","(?<link>[^"]+)/gm; 

  return [...categoryContent.matchAll(artistsPattern)].map(({ groups }) => ({
    ...
  }));

```

To make this array we need to [destructure](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring%5Fassignment) `groups` and define object with artist info, that contains `artist`, `works` , `thumbnail` and `link` fields. To make valid `link` we need to parse some [encoded symbols](https://en.wikipedia.org/wiki/List%5Fof%5FUnicode%5Fcharacters) (e.g. "\\u003d") with [JSON.parse()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global%5FObjects/JSON/parse) method:

```javascript
    artist: groups.artist,
    works: groups.works,
    thumbnail: `https:${groups.thumbnail}`,
    link: `${AXIOS_OPTIONS.baseURL}${JSON.parse(`"${groups.link}"`)}`,

```

Then, we write a function that makes the request and returns the received data. We received the response from [axios](https://www.npmjs.com/package/axios) request that has `data` key that we [destructured](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring%5Fassignment) and return it:

```javascript
function getArtistsInfo() {
  return axios
        .get("/category/artist", AXIOS_OPTIONS)
        .then(function ({ data }) {
    ...
  });
}

```

In this function we declare the `results` object, and using different RegEx patterns add new keys with results from each category:

```javascript
const results = {};

    //https://regex101.com/r/kyTsgs/1
const popularCategoryPattern = /"PopularAssets:(?<content>.+?)\["stella\.pr/gm;
[...data.matchAll(popularCategoryPattern)]
    .forEach(({ groups }) => (results.popular = getResultsFromCategory(groups.content)));

    //https://regex101.com/r/u1KZFf/1
const azCategoryPattern = 
        /"(?<letter>[^"])",\["stella\.pr","(?<content>.+?)[\w"||\d]\]{2,3},\[/gm;
[...data.matchAll(azCategoryPattern)]
    .forEach(({ groups }) => (results[groups.letter] = getResultsFromCategory(groups.content)));

    //https://regex101.com/r/5JHaQB/1
const timeCategoryPattern = 
        /\[{1,2}"(?<time>[^"]{3,8})","?\w{4,7}.+?\["stella\.pr","DatedAssets(?<content>.+?)"?\d{3,5}"\]/gm;
[...data.matchAll(timeCategoryPattern)]
    .forEach(({ groups }) => (results[groups.time] = getResultsFromCategory(groups.content)));

return results;

```

And finally, we need to run our function and print all the received information in the console with the [console.dir](https://nodejs.org/api/console.html#consoledirobj-options) method, which allows you to use an object with the necessary parameters to change default output options:

```javascript
getArtistsInfo().then((result) => console.dir(result, { depth: null }));

```

Now we can launch our parser:

```bash
$ node YOUR_FILE_NAME # YOUR_FILE_NAME is the name of your .js file

```

## Output

```javascript
{
   "popular":[
      {
         "artist":"Vincent van Gogh",
         "works":"340",
         "thumbnail":"https://lh3.googleusercontent.com/nizOQO4H8v1F3-2Do1m9Rj1j4baLBqLS57HMbyi83_kZv_F1yFEc-2iqHYqBRGzmjg",
         "link":"https://artsandculture.google.com/entity/vincent-van-gogh/m07_m2?categoryId=artist"
      },
      {
         "artist":"Claude Monet",
         "works":"283",
         "thumbnail":"https://lh3.googleusercontent.com/hML980F4Qz51OPJqearbH_aPSxUPGFC7MaPEgYaxaG1CVdh8eh25aMtY67XkP6ZbnA",
         "link":"https://artsandculture.google.com/entity/claude-monet/m01xnj?categoryId=artist"
      },
      ...and other popular results
   ],
   "A":[
      {
         "artist":"A. B. Frost",
         "works":"5",
         "thumbnail":"https://lh3.ggpht.com/Lfwsu29qks8oAArsSnIrMYXCyAW1eJHSs_zRtV87_kuGOj31LZfabjT14QEg4g",
         "link":"https://artsandculture.google.com/entity/a-b-frost/m06b7cg?categoryId=artist"
      },
      {
         "artist":"A. J. Casson",
         "works":"13",
         "thumbnail":"https://lh3.googleusercontent.com/oPxgz35wxodv8998Nsarup0c78_gOey6FoR9BS2oHm303-g3F_I3yrjD9GooE8IQ5-k",
         "link":"https://artsandculture.google.com/entity/a-j-casson/m0695mj?categoryId=artist"
      }
      ...and other "A" results
   ],
   ...and other "A-Z" results
   "Far Past":[
      {
         "artist":"Titian",
         "works":"141",
         "thumbnail":"https://lh3.googleusercontent.com/uwSkz-wNQt6ts1mgb0GNgUpllFY_4I6Pa7W5XrAHkzVPiPZeekbS6KLEjvL2OJB41QA",
         "link":"https://artsandculture.google.com/entity/titian/m0144mv?categoryId=artist"
      },
      {
         "artist":"Sandro Botticelli",
         "works":"63",
         "thumbnail":"https://lh3.googleusercontent.com/M7LhlnWSe_9NS4IMW62GWpAJcxnEtBd2vjwDGuEVz3P0Qp-7IpASzVvXRFE-HBHL",
         "link":"https://artsandculture.google.com/entity/sandro-botticelli/m0jr3g?categoryId=artist"
      },
      ...and other "Far Past" results
   ],
   ...and other "Time" results
}

```

If you want to see some projects made with SerpApi, [write me a message](mailto:miha01012019@gmail.com).

---

Join us on [Twitter](https://twitter.com/serp%5Fapi) | [YouTube](https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg%5Fmg)

Add a [Feature Request](https://github.com/serpapi/public-roadmap/issues)💫 or a [Bug](https://github.com/serpapi/public-roadmap/issues)🐞