How to use cURL in Javascript (and its alternative)

If you like cURL and use JavaScript (Nodejs) daily, you might wonder how to use cURL in your JavaScript programs. While it's possible to use the libcurl library in JavaScript, there are also other alternatives for achieving the same goal without using cURL.

Read this post if you're interested in using cURL for web scraping how to utilize cURL for web scraping and its basic commands

How to use cURL with Javascript?

To use cURL in Javascript, we can use exec method from shell command or a 3rd party library called node-libcurl .

As mentioned, we'll also cover alternatives to cURL. But first, let's see how to use cURL directly.

Option A: Using shell command
Since cURL is a command line, we need to learn how to execute a command in Node.js (Javascript), which uses the child process.

Here is the sample code:

const { exec } = require('child_process');

exec('curl -s https://example.com', (error, stdout, stderr) => {
  if (error) {
    console.error(`Error: ${error}`);
    return;
  }
  if (stderr) {
    console.error(`stderr: ${stderr}`);
    return;
  }
  console.log(`stdout: ${stdout}`);
});

We add the -s (silent) to make cURL silent, which means it won't show the progress meter or error messages. If you still want to see error messages, you can use -S. This should help make the output cleaner and more focused on the actual data you're interested in.

Option B: Using node-libcurl package

There is a Nodejs package called node-libcurl . This package provides a binding to the libcurl library.

GitHub: https://github.com/JCMais/node-libcurl

How to use node-libcurl?

Install the node-libcurl library

npm i node-libcurl --save // or yarn add

A simple request with async/await method

const { curly } = require('node-libcurl');

async function run() {
  const { statusCode, data, headers } = await curly.get('https://www.google.com')
  
  console.log(statusCode)
  console.log('---')
  console.log(data)
}

run();

A JSON post example:

const { curly } = require('node-libcurl')
const { data } = await curly.post('http://httpbin.com/post', {
  postFields: JSON.stringify({ field: 'value' }),
  httpHeader: [
    'Content-Type: application/json',
    'Accept: application/json'
  ],
})

console.log(data)

cURL alternative in Javascript client side

If you want to use cURL inside your HTML code, a.k.a. running it in a browser, you can use an AJAX request or Fetch API.

This is because cURL, as a command-line tool, cannot be embedded directly into HTML or run in a browser environment. Instead, AJAX and Fetch API provide a way to make HTTP requests similar to cURL but within the context of a web page.

These APIs allow you to send and receive data asynchronously without reloading the page, enabling interactions with servers in a way that's integrated with your browser-based JavaScript code.

Ajax request example:

var xhr = new XMLHttpRequest();
xhr.open("GET", "http://example.com/api/data", true);
xhr.onreadystatechange = function() {
  if (xhr.readyState === 4 && xhr.status === 200) {
    console.log(xhr.responseText);
  }
};
xhr.send();

Fetch API example:

fetch('http://example.com/api/data')
  .then(response => {
    if (!response.ok) {
      throw new Error('Network response was not ok');
    }
    return response.json();
  })
  .then(data => console.log(data))
  .catch(error => console.error('Fetch error:', error));

You can use fetch API for more convenient syntax. It's already supported in the majority of browsers: https://caniuse.com/mdn-api_fetch

cURL alternative on the Javascript server side (Node.js)

You probably also need to run cURL on the server side of your code using Node.js. Don't worry; we have many alternatives to make HTTP requests on the server side. There are:

• Axios
• Got
• node-fetch
• request
• SuperAgent
• built-in HTTP module
• and more!

Let's say we pick Axios as the alternative. Make sure to install it first with npm i axios --save

Here are a few code samples:

Simple Get request

import axios from 'axios';

const response = await axios.get('http://example.com');

It's equivalent to: curl example.com

I'll skip the import for the next samples

Saving raw HTML results well in a file

Since we're writing in a programming language, we now have the flexibility to mix and match this code with other actions as well.

const axios = require('axios');
const fs = require('fs');

Axios.get('https://example.com')
    .then(response => {
        fs.writeFile('index.html', response.data, (err) => {
            if (err) {
                console.error('Error writing file:', err);
            } else {
                console.log('File saved as index.html');
            }
        });
    })
    .catch(error => {
        console.error('Error fetching URL:', error);
    });

Equivalent to curl example.com -o "index.html"

Send a POST request with data

const axios = require('axios');

const postData = {
    param1: 'value1',
    param2: 'value2'
};

axios.post('http://example.com', postData)
    .then(response => {
        console.log('Response:', response.data);
    })
    .catch(error => {
        console.error('Error:', error);
    });

Equivalent to: curl -d "param1=value1&param2=value2" -X POST http://example.com

Basic HTTP Request with auth

const axios = require('axios');

Axios.get('http://example.com', {
    auth: {
        username: 'username',
        password: 'password'
    }
})
.then(response => {
    console.log('Response:', response.data);
})
.catch(error => {
    console.error('Error:', error);
});

Equivalent to: curl -u username:password http://example.com

You can see all the supported configs for Axios on this page: https://axios-http.com/docs/req_config

Adding a custom header

const axios = require('axios');

axios.get('http://example.com', {
    headers: {
        'X-Custom-Header': 'value'
    }
})
.then(response => {
    console.log('Response:', response.data);
})
.catch(error => {
    console.error('Error:', error);
});

Equivalent to: curl -H "X-Custom-Header: value" http://example.com

Simulate using a user agent

Like above, user agent information can be attached to the headers.

const axios = require('axios');

Axios.get('http://example.com', {
    headers: {
        'User-Agent': 'User-Agent-String'
    }
})
.then(response => {
    console.log('Response:', response.data);
})
.catch(error => {
    console.error('Error:', error);
});

Equivalent to: curl -A "User-Agent-String" http://example.com

Using proxies

const axios = require('axios');

axios.get('http://example.com', {
    proxy: {
        host: 'proxyserver',
        port: port // Replace 'port' with the actual port number
    }
})
.then(response => {
    console.log('Response:', response.data);
})
.catch(error => {
    console.error('Error:', error);
});

Equivalent to: curl -x http://proxyserver:port http://example.com

Tips

You can use an online tool to convert some of your curl commands into a Javascript command with https://curlconverter.com/

What's the benefit of using Axios instead of executing cURL in Nodejs?

Some of the benefits of using Axios instead of cURL are:

1. Programming Interface:
   • Axios: Offers a JavaScript programming interface, allowing integration with Node.js code, error handling, and data manipulation more seamlessly.
   • cURL: A command-line tool, not inherently part of JavaScript. Requires additional handling to integrate with Node.js, typically through child processes.
2. Asynchronous JavaScript:
   • Axios: Supports promises and async/await, aligning well with modern JavaScript practices.
   • cURL: Being a command-line tool, it doesn’t naturally fit into the asynchronous JavaScript model and requires extra work to handle asynchronously.
3. JSON Support:
   • Axios: Automatically handles JSON data, including parsing of JSON responses and stringifying JSON requests.
   • cURL: Does not inherently handle JSON formatting or parsing; requires manual handling.

If you want to learn more about how to implement a web scraping in javascript, feel free to read our other post.

I hope this helps. See you in the next post!