CSE 154

Lecture 19: More async/await, glob, and API Documentation

Agenda

Review async/await

Finding files/directories with glob

API documentation

Check Your Understanding: Callbacks and Promises

What is the difference between a callback and a Promise?
Why are callbacks and Promises useful?

Check Your Understanding: Node.js APIs

What files should you ignore when pushing to Gitlab in a Node project?
Where does package.json come from, and when is it updated?
What is the purpose of a public directory and where is it relative to app.js?
What is an example of a core module and a non-core module?

More about example project directories

Check Your Understanding: File I/O in Node.js

What module do we use for file processing, and what are some of its functions?
What common properties of all fs functions?
What are the different types of errors we should consider in an API that uses client requests and file processing?

File I/O So Far

Read files with fs.readFile

Write files with fs.writeFile/fs.appendFile

Read directories with fs.readdir

Pre-Check Question

What's the output?

// this function intends to log active users to a single file.
// in this case, we are worried about the order, so ensure it is preserved.
function logUsers(clientList) {
  for (let client of clientList) {
    fs.writeFile("user-log.txt", client, "utf8", (err) => {
      if (err) {
        console.error(err + "\nFailed to log user.");
      }
    });
  }
}

logUsers(["Tal", "Manny", "Hudson"]);

Demo code

What's the Issue?

First, if we want to append each name, we should use fs.appendFile instead of fs.writeFile

That still won't solve the problem of order though

This is an example where async/await come in handy!

Using async/await with File I/O

There are a few ways we can use async/await with fs, but the key thing is we need to use await on functions that return Promises

fs.readFile and fs.writeFile are asynchronous but do not return Promises

We could make our own Promise-returning functions, but that can be tedious.

Instead, we'll need to "promisify" these functions so we can use async and await

"Promisifying" Functions

The core utils package comes with a function promisify which we can use to return a Promise-returning version of a function

Let's practice with the previous example (and update it to use fs.appendFile).

A Better Solution

const fs = require("fs");
const util = require("util");

async function logUsers(clientList) {
  const appendFile = util.promisify(fs.appendFile);
  for (let client of clientList) {
    try {
      await appendFile("user-log.txt", client + "\n");
    } catch (err) {
      console.error(err + "\nFailed to append file with user: " + client);
    }
  }
}

logUsers(["Tal", "Manny", "Hudson"]);

Demo code

Comparison: `readFile` 3 Ways

The following functions are three ways to read a file. Note that all are good solutions, but the more asynchronous fucntions you work with in your functions, the more useful it is to use async/await with promisified functions.

const fs = require("fs");
const util = require("util");
const readFile = util.promisify(fs.readFile);

function readFileCallback() {
  // Syntax: readFile(fileName, (error, result) callback)
  fs.readFile("data/example.txt", "utf8", (err, contents) => {
    if (err) {
      handleError(err);
    } else {
      printContents(contents);
    }
  });
}

// Promisified version
function readFilePromisified() {
  const readFilePromise = readFile("data/example.txt", "utf8");
  readFilePromise
    .then(printContents)
    .then(() => { console.log("Done reading file!") })
    .catch(handleError);
}

async function readFileAsync() {
  try {
    const contents = await readFile("data/example.txt", "utf8");
    printContents(contents);
    console.log("Done reading file!");
  } catch (err) {
    handleError(error);
  }
}

function printContents(contents) {
  console.log("File contents:");
  console.log(contents);
}

function handleError(err) {
  console.log("There was an error: " + err);
}

readFileCallback();
readFilePromisified();
readFileAsync();

Error-handling with `async`/`await`

For error-handling with async/await, you must use try/catch instead of .then/.catch

The catch statement will catch any errors that occur in the then block (whether it’s in a Promise or a syntax error in the function), similar to the .catch in a fetch promise chain

Remember that if you are using these in a web service, you should handle client-specific (400) errors differently than server-specific (500) errors (such as those caught by a fs function).

Recall: Reading Directories

fs.readdir returns an array of files within a given directory path

fs.readdir("data", (err, paths) => {
  if (err) {
    console.error("Error reading directory");
  } else {
    console.log("data directory contents: ");
    console.log(contents);
  }
});

JS (standard callback version)

const readdir = utils.promisify(fs.readdir);
try {
  const paths = await readdir("data");
  console.log("data directory contents: ");
  console.log(paths);
} catch (err) {
  console.error("Error reading directory");
}

JS (async/await version)

Example code: directory-reading.js

Finding Files in Node.js

Sometimes you will want to find files and directories in your Node.js programs

This is useful particularly when writing scripts, as well as extensible APIs that rely on directory structures (e.g. returning a list of all menu categories based on non-empty directories)

What if we could match patterns to look for certain files and directories?

We can't with fs, but there are various modules to help

How do we pick a new module to use in our Node.js projects?

Choosing Modules

There are many modules we can use in Node.js in addition to the core modules

"glob" is a method that many programming languages use to process the file system with regex-like patterns

Node.js has many glob modules

A very useful site to choose modules is npmtrends.com

We will use glob since it has most support and is very easy to get started with.

Using `glob`

glob(pattern, [options], (err, matches) => { ... });

JS (syntax)

glob("*.png", (err, matches) => {
  if (err) {
    console.error("There was an error: " + err);
  } else {
    console.log(".png images in current directory:", matches);
  }
});

Check out some useful documentation of more features here

Examples: glob-examples.js

Promisifying Glob

Since glob takes a callback as its last argument, we can use utils.promisify to define a promise-returning version.

const globPromise = util.promisify(glob);

async function globAsync() {
  try {
    let matches = await globPromise("data/*");
    console.log(".png images in current directory:", matches);
  } catch(err) {
    console.error("There was an error: " + err);
  }
}

A More Motivating Example

Often, web services don't rely on hard-coding data in files

For example, our menu web service returns JSON, but what if we want to support different JSON responses?

{
  "categories" : {
    "Drinks" : [
      {
        "name" : "Classic coffee",
        "description" : "The classic.",
        "image" : "coffee.png",
        "in-stock" : true
      }, ...
    ],
    "Foods" : [
      ...
    ]
 }

We might want to keep track of item quantity for example, and uses that to determine the in-stock key

Let's take a closer look at the new directory structure, and get a bit more practice with glob

Example with Cafe Directory

The cafe web service now holds its data in a directory structure to help process the results

cse154-cafe/
   app.js
   categories/
     Drinks/
       bubble-tea/
         info.txt
         purchase-history.txt
       classic-coffee/
         info.txt
         purchase-history.txt
       ...
     Foods/
       ...
   public/
     fetch-menu.js
     admin.js
     img/
   stock-img/
     ...

Contents (abbreviated)

let categoryPaths = await globPromise("categories/*");
// ["categories/Foods", "categories/Drinks"]

let publicFolders = await globPromise("public/*/");
// ["public/img/"]

let drinksInfo =
  await globPromise("categories/Drinks/*/info.txt");
// ["categories/Drinks/bubble-tea/inf.txt",
//  "categories/Drinks/classic-coffee/info.txt",
//  "categories/Drinks/the-sippy/info.txt"]

// ** recursively searches within the current directory
// and all subdirectories
let allJS = await globPromise("**/*.js");
// ["app.js", "public/fetch-menu.js", "public/admin.js"]

// can use [patt1|patt2|...] to match patt1 or patt2 or ...
let allImages = await globPromise("**/*.[png|jpg]");

Options

Like many other fs functions, there is an optional "options" parameter to the glob function to support various useful options

To use, just pass an object as the second argument with the option(s) you want as keys.

// the nodir option ignores directories
let dataFiles = await globPromise("data/*", { "nodir" : true });

// the ignore option takes patterns to ignore
let noGifs = await globPromise("public/img/*", { "ignore" : ['*png'] });

API Documentation

Some Good Examples

Deck of Cards API
Stripe API
NASA APOD API
Spotify API
Scryfall (Magic Card Game)
REST Countries
TMDB
placekitten - easy to get started, very limited on response details (e.g. errors)

API Documentation in this Class

At minimum, all route functions must be documented (1-2 sentences) and your overall file header should include a summary of all endpoints (with their response formats/possible errors)

If you have API documentation, it can be much easier to refer to in your header comments

In HW4, you will not be submitting an APIDOC.md, so your file header should serve as an overview of the endpoints

In CP4 and the Final Project, you are required to submit APIDOC.md

Writing APIDOC.md

There are provided templates (APIDOC.md), but you are free to modify it based on what you like most.

If you would like to use an API documentation tool generator (such as POSTMan or Swagger), you are free to do so, but must receive permission from the instructor to receive credit in your submission in place of APIDOC.md.

We'll get more practice in section tomorrow!

Looking Ahead

Tomorrow, you will get practice with using file-processing, async/await, glob, and API documentation.

On Friday, we will learn how to handle POST requests and how to deploy our Node APIs on a real server!