How to use HTTPS module in Node.js programs?

Sending, receiving and processing data with HTTPS module in Node.js programs

Raywon Kari - Published on December 26, 2020 - 7 min read

Recently I was working on a task which involves sending an API request to an external URL, and then using the response for further processing. I was working with Node.js in that task.

As I have recently started to raise the bar when it comes to my programming skills, it was obvious that I struggled a bit to implement this task, especially on the part where we need to wait until the external API responds, capture it and to do further processing of that data.

One of the things I spent quite some time reading and trying to understand is:

• how to send requests using HTTPS module available by default in Node.js
• how to capture the response from those API calls
• how to further process that response.

In this blog post, I am going to share these learnings.

Before diving deep in to the concepts, let's first take a quick look at Synchronous, Asynchronous programming & Promises.

Sync & Async:

Synchronous programming is something, where a set of tasks are executed one after the other. In asynchronous programming, several tasks can be executed in the background. These are also termed as blocking and non-blocking methods.

One common example often discussed in the internet is that of going to a movie theatre and standing in a queue to purchase tickets, and that of going to a restaurant and ordering food. Buying tickets is synchronous, since people have to wait until the ones in front are done, whereas in the restaurant, everyone can place orders and eat at the same time.

Promises:

According to wikipedia, a promise is a commitment by someone to do or not do something. It is exactly the same context in the programming world.

A promise is an object, which represents a particular data which is collected eventually, when a particular asynchronous operation is completed. It can be a successful call or an error. The following diagram shows both the use cases.

Resolve corresponds to the data received when the Promise is fulfilled, whereas reject corresponds to the data when the Promise is rejected.

It can also be demonstrated in the following format:

// This function returns a Promise.

async function returnPromise() {
  return new Promise((resolve, reject) => {
    if (someCondition) {
      resolve(data)
    } else {
      reject(error)
    }
  })
}

This Promise (.then .catch) transforms into the following format:

function(data) {
  // use data returned from Promise
  // this function is equivalent to .then
}

function(error) {
  // Handle the error
  // this function is equivalent to .catch
}

I created a small example in a git repo here. Clone it, and have a look at the example to see the whole picture.

# https://github.com/raywonkari/nodejs-https-examples
> cd promise-example/
> node index.js

This way we can use Promises when we want to wait until a request is fulfilled before going forward. Some of the use cases include, when calling an external/internal API, when we want to trigger/wait for an event to occur and so on.

Basic definition of HTTPS is the HTTP protocol over TLS/SSL. In Node.js, this is implemented as a separate module of its own. Full documentation can be found here.

In this section, let's checkout:

• How to send requests
• How to capture the response

Sending Requests

Depending on which type of request we want to make, the necessary input parameters can change. All of the request methods can be found here.

Let's try querying GitHub's public API using a GET request. We need the following components:

• Options
• Sending request
• Printing the result

Here is an example:

const https = require("https");

const options = {
  hostname: "api.github.com",
  path: "/users/raywonkari",
  method: "GET",
  headers: {
    Accept: "application/vnd.github.v3+json",
    "User-Agent": "raywon-nodejs-example-app",
  },
};

const req = https.request(options, (res) => {
  console.log("statusCode:", res.statusCode);
  console.log("headers:", res.headers);

  res.on("data", (d) => {
    console.log(d.toString());
    // Capture "d" and do further processing.
  });
});

req.on("error", (e) => {
  console.error(e);
  // Catch the error, handle it.
});
req.end();

I added this example in the git repo here. To run this, simply do node get.js. We will either get the data or the error as the response from GitHub's public API.

In this last section, we will go through the steps needed, in order to capture the response and process it later. The example above, is synchronous in nature, meaning when we execute the example, we are sending a GET request, waiting until we get a response back, and then we are closing the request.

This will not work when we run this in an asynchronous program because in async programs, several tasks will be executed in the background, and we need a mechanism in place to say that, we will need to capture the response and process that later instead of waiting for it, because async programs cannot wait, but should handle the response later.

That's where Promises come into the picture. We use Promises, to continue executing the GET request in background, and once the request is processed, and response is received, then we capture that data.

Here are the steps needed:

1. Return a new promise with resolve and reject, which contains the data and the error object.
2. Capture that result using one of .then() .catch() or await.

Example

const https = require("https");

const options = {
  hostname: "api.github.com",
  path: "/users/raywonkari",
  method: "GET",
  headers: {
    Accept: "application/vnd.github.v3+json",
    "User-Agent": "raywon-nodejs-example-app",
  },
};

console.log("Capturing the response using .then and .catch");
getRequest()
  .then((data) => {
    console.log(data);
  })
  .catch((error) => {
    console.log(error);
  });

(async () => {
  let dataResponse = await getRequest();

  console.log("Capturing response using await");
  console.log(dataResponse);
})();

async function getRequest() {
  return new Promise((resolve, reject) => {
    const req = https.request(options, (res) => {
      let body = "";

      // we will receive chunks of data in this case, 
      // therefore we need to concatenate the data
      res.on("data", (chunk) => {
        body += chunk;
      });

      res.on("end", () => {
        resolve(body);
      });
    });

    req.on("error", (e) => {
      reject(e);
    });

    req.end();
  });
}

• If you followed along, we used two ways of capturing the result i.e., with .then() .catch() and using await. Both are valid, but each method has its own purpose.
• .then() .catch() is used in sync programs to capture Promises and handle them.
• await is used in async programs to capture promises and handle them.
• There is another interesting thing in the code, i.e., ( async() => {...})(). This is known as Immediately Invoked Function Expression (IIFE). This a way to run async programs in a sync program.
• If you are developing a similar code which is executed in an async environment such as AWS Lambda, you wouldn't need to use the IIFE but write the await calls directly. Let's keep these concepts for another blog post :)

All of the examples are in the git repo here.