CSE 154

Lecture 22: Glob, Post, and Middleware

Agenda

• Recap Glob
• Promises

Reading Directories

fs.readdir returns an array of files within a given directory path

const fs = require('fs').promises;
try {
  const paths = await fs.readdir("data");
  console.log("data directory contents: ");
  console.log(paths);
} catch (err) {
  console.error("Error reading directory");
}

JS (async/await version)

Example code: directory-reading.js

Using glob

glob(pattern, [options], (err, matches) => { ... });

JS (syntax)

glob("*.png", (err, matches) => {
  if (err) {
    console.error("There was an error: " + err);
  } else {
    console.log(".png images in current directory:", matches);
  }
});

JS

Check out some useful documentation of more features here

Examples: glob-examples.js

Question:

What's the difference between Glob and Readdir?

Questions:

1. What happens when fs.readdir doesn't find anything?
2. What happens when glob doesn't find anything?

Promisifying Glob

Since glob takes a callback as its last argument, we can use utils.promisify to define a promise-returning version.

const utils = require('utils');
const globPromise = utils.promisify(glob);

async function globAsync() {
  try {
    let matches = await globPromise("data/*");
    console.log(".png images in current directory:", matches);
  } catch(err) {
    console.error("There was an error: " + err);
  }
}

JS

A More Motivating Example

Often, web services don't rely on hard-coding data in files

For example, our menu web service returns JSON, but what if we want to support different JSON responses?

{
  "categories" : {
    "Drinks" : [
      {
        "name" : "Classic coffee",
        "description" : "The classic.",
        "image" : "coffee.png",
        "in-stock" : true
      }, ...
    ],
    "Foods" : [
      ...
    ]
 }

We might want to keep track of item quantity for example, and uses that to determine the in-stock key

Let's take a closer look at the new directory structure, and get a bit more practice with glob

Example with Cafe Directory

The cafe web service now holds its data in a directory structure to help process the results

cse154-cafe/
   app.js
   categories/
     Drinks/
       bubble-tea/
         info.txt
         purchase-history.txt
       classic-coffee/
         info.txt
         purchase-history.txt
       ...
     Foods/
       ...
   public/
     fetch-menu.js
     admin.js
     img/
   stock-img/
     ...

let categoryPaths = await globPromise("categories/*");
// ["categories/Foods", "categories/Drinks"]

let publicFolders = await globPromise("public/*/");
// ["public/img/"]

let drinksInfo =
  await globPromise("categories/Drinks/*/info.txt");
// ["categories/Drinks/bubble-tea/inf.txt",
//  "categories/Drinks/classic-coffee/info.txt",
//  "categories/Drinks/the-sippy/info.txt"]

// ** recursively searches within the current directory
// and all subdirectories
let allJS = await globPromise("**/*.js");
// ["app.js", "public/fetch-menu.js", "public/admin.js"]

// can use [patt1|patt2|...] to match patt1 or patt2 or ...
let allImages = await globPromise("**/*.[png|jpg]");

Options

Like many other fs functions, there is an optional "options" parameter to the glob function to support various useful options

To use, just pass an object as the second argument with the option(s) you want as keys.

// the nodir option ignores directories
let dataFiles = await globPromise("data/*", { "nodir" : true });

// the ignore option takes patterns to ignore
let noGifs = await globPromise("public/img/*", { "ignore" : ['*png'] });

Choosing Error Codes

Use 400 (Invalid Requests) for client-specific errors.

• Invalid parameter format (e.g. "Seattle" instead of a required 5-digit zipcode)
• Missing a required query parameter
• Requesting an item that doesn't exist

Use 500 (Server error) status codes for errors that are independent of any client input.

• Errors caught in fs or glob that are not related to any request parameters
• SQL Database connection errors (next week!)
• Other mysterious server errors...

Middleware

POST Parameters

With GET endpoints, we've used req.params and req.query to get endpoint parameters passed in the request URL.

But remember that POST requests send parameters in the Request body, not in the URL.

app.post("/contact", (req, res) => {
  let name = req.params.name; // this doesn't work!
  ...
});

JS

What is a disadvantage of sending parameters in a URL?

• Not secure, limit to the length of data we can send in a URL

So, how can we get POST parameters sent by a client?

Handling Different POST Requests

POST requests can be sent with different data types:

• application/x-www-form-urlencoded
• application/json
• multipart/form-data

In Express, we have to use middleware to extract the POST parameters from the req.body. For the first two, there is built-in middleware - we don't need middleware for text/plain.

With forms and fetch, we use the FormData object to send POST parameters, which is always sent as multipart/form-data.

There is no built-in middleware to access the req.body params for multipart content.

Another module!

The multer Module

A module for extracting POST parameter values sent through multipart POST requests like those sent with FormData

Has a lot of functionality to support file uploading, but we will just use it to access the body of a POST request sent through FormData, which we can't get with just req.body.

To use, we'll need to set an option to ignore upload features with multer().none()

const multer = require("multer");

app.use(multer().none());

JS

Remember to run npm install multer in any project that uses it.

Supporting all POST requests

We often don't want to make assumptions about what method a client uses to POST data. It's best to support all three with the appropriate middleware.

// other required modules ...
const multer = require("multer");

// for application/x-www-form-urlencoded
app.use(express.urlencoded({ extended: true })) // built-in middleware
// for application/json
app.use(express.json()); // built-in middleware
// for multipart/form-data (required with FormData)
app.use(multer().none()); // requires the "multer" module

...
app.post("/contact", (req, res) => {
  let name = req.body.name;
  let email = req.body.email;
  let message = req.body.message;
  let timestamp = new Date.toUTCString();
  // validate parameters, then update message.json file with new data
  ...
});

JS

Summary of Handling a POST Request

1. Use app.post instead of app.get
2. Use req.body.paramname instead of req.params.paramname/req.query.paramname
3. Require the multer (non-core) module with the rest of your modules
4. Use the three middleware functions to support the three different types of POST requests from different possible clients
5. Test your POST requests with Postman or with fetch and FormData in client-side JS (similar to HW3) - remember you can't test POST requests in the URL!

API Documentation

Some Good Examples

• Deck of Cards API
• Stripe API
• NASA APOD API
• Spotify API
• Scryfall (Magic Card Game)
• REST Countries
• TMDB
• placekitten - easy to get started, very limited on response details (e.g. errors)

API Documentation in this Class

At minimum, all route functions must be documented (1-2 sentences) and your overall file header should include a summary of all endpoints (with their response formats/possible errors)

If you have API documentation, it can be much easier to refer to in your header comments

Writing APIDOC.md

There are provided templates (APIDOC.md), but you are free to modify it based on what you like most.

If you would like to use an API documentation tool generator (such as POSTMan or Swagger), you are free to do so, but must receive permission from the instructor to receive credit in your submission in place of APIDOC.md.

We'll get more practice in section tomorrow!