Section 14: Glob and Readdir

Section Goals

By the end of this section, you should know how to:

  • Use glob and readdir to find and manipulate files in a directory
  • Document API code for clients

Reminder:

The final project milestone deadline has been extended to the 14th. There are no late days allowed on this so be sure to get it done! You will find the first exercise today good practice for the milestone.

Searching directories with glob

In the last section, we hard-coded file names for scores.json and dictionary.txt. However, there are many situations when we want to grab lots of files, or when we are not sure what the files are called. This is where glob comes in handy.

To use glob, we must import the glob package. 

const util = require("util");
const glob = require("glob");
const globPromise = util.promisify(glob); // override to get a promise version

After this, you can use glob to retrieve arrays of files or directories. Glob takes one argument, the path of the files you want, and it returns a promise whose value is equal to the array of matching files. 

let scores = await globPromise("scores/*.json");
let firstScore = await readFilePromise(scores[0], "utf8")

The * can match any non-slash character, meaning you can find files with certain formats within a directory. After this line of code, scores will contain an array of file paths to these JSON files, each of which you could plug into readFile to get the contents of the JSON.

Searching directories with readdir

readdir will do about the same thing as glob, with some minor differences. First, you will want to "promisify" it as we do with readFile. 

const fs = require("fs");
const util = require("util");
const readdir = util.promisify(fs.readdir);

Glob takes as an argument a pattern for the files, using stars for uncertain parts of the path, and returns full file paths. readdir (promisified) takes the path of a directory, and returns the filenames of the files within. These file names do not include the path to the file. 

let scores = await readdir("scores/");
let firstScore = await readFile("scores/" + scores[0], "utf8")

Exercise 1: API Documentation

A programmer's job is never done when the code starts working. Working code is great, but not if people don't know how to use it!

The following are examples of different types of API documentation, some better than others

What do you think is important to include in API documentation? What makes API documentation bad?

Writing APIDOC.md

Included with Tuesday's starter code was an APIDOC.md. Fill it in with details about the API and how to use it. Do not include implementation details. This sort of file is intended for people trying to use your API endpoints, telling them how to structure their requests and what kinds of responses they will get back.

As a handy tool, you can preview what your markdown file looks like by right clicking on the file in atom and selecting "Markdown Preview".

You will be writing similar files for your projects! For assignments that do not have an APIDOC.md (e.g. HW4), please include similar documentation as a file header comment.

Solution

Sample solution documentation

Exercise 2: Hybrids!

In this exercise, you will use glob to display photos of... animals. You can find the starter files and images in this hybrids.zip folder.

Here is a solution to reference the expected behavior, but don't peek at the code! hybridsSOLN.zip.

Slides with more details are provided below.

Overview (1/3)

The hybrids.html page uses hybrids.js (completed for you) to make GET requests to our webservice, and display images on the page using the plain text results. When the #submit-one button is clicked, a request will be made to /:animal where the value of :animal is whatever the user input into the #animal text input box.

If the user clicks on #submit-all instead, a request will be made to /animals/all. The response will then be used to populate the #results area with the resulting images.

First Endpoint (2/3)

Write a /:animal GET endpoint that uses glob so that it will return a plain text result of all images containing the string :animal (each on their own line). Use image file paths from public/images/. Remember that the * in glob matches 0 or more of any character. The starter code promisifies glob for you with globPromise so you can easily use async/await.

Second Endpoint (3/3)

Next, write a /animals/all endpoint that does the same thing as the first endpoint, but provides all images.

It is important to note that the relative path to the images from public/hybrids.html is different than the path from hybrids-app.js. So, for the image paths you pass back to the client, you will need to strip off the public/ from the front of the path.

It's also important to avoid extra new lines at the end of the output, since those will lead to improperly rendered images. You can use the String trim method to remove trailing whitespace.

Exercise 3: Recipe Generator

It's always fun to try new things. When it comes to cooking, sometimes it's hard to find that spark of inspiration. In this exercise, you'll use folder and file processing to create a web service that takes in a single parameter name and outputs a randomly-generated recipe idea based on the letters in the name.

The first letter of the name will correspond to the first letter of the recipe name, and the rest of the letters of the name will each correspond to a randomly-generated ingredient for that recipe. The final recipe will be output in plain text.

Go "down" to work through the slides specific to this exercise.

Exercise 3: Provided Files

Download and unzip the provided recipe-generator.zip. In this folder, there are two sub-directories, foods and ingredients. foods contains exactly 26 txt files each corresponding to a lower-cased letter of the English alphabet, listing recipe names. ingredients includes many files consisting of a letter followed by a number, each file listing a single ingredient.

There is also a starting recipe-generator.js provided which you will fill in to create one /:name endpoint.

Exercise 3: Example file in foods


Macaroni
Manicotti
Mantou
Marmalade
Masala
Milkshake
Minestrone Soup
Mochi
Mooncake
Muffin

foods/m.txt contents

Note that there is likely a blank line at the end of the file. Since you don't want an empty entry when splitting the foods up, we recommend using the trim method of Strings to trim the whitespace from the end.

Exercise 3: Example file in ingredients


1 cup rolled Oats

ingredients/o1.txt contents


1 Okra

ingredients/o2.txt contents

Note that each ingredient file contains a possible ingredient starting with the corresponding letter. To help generate a "reasonable" recipe, each ingredient option includes a unit (e.g. for the ingredient "Oreo" in ingredients/o.txt, it may be listed as "1 Oreo", but there is also an option for "1 box of Oreos" for extra oreo-ness in your recipe). Your code should not depend on this format though, it just makes the output nicer.

(hint: You will need glob to sort through these)

Exercise 3: Example Output

The following is an example (random) output for a request to
/Mowgli: 


Mowgli's Muffin
Directions:
In a bowl, mix:
1 Oreo
1 gallon of Water
1 oz of Green peas
1 Lentil
1 Ice cube

Cook for 6 minutes and serve!

Example output (plain text)

Exercise 3: Generating and Outputting the Recipe

Now that we know the format of the files we have in the two directories, let's implement the recipe generator.

Implement a /:name GET endpoint that takes :name as a URL parameter. It should not matter if the name has capital letters or not. This endpoint will send a plain text response with the recipe in the format outlined on the next slide.

Exercise 3: Output Format

  1. The first line is :name's recipeName, where the recipe name is randomly chosen from a food beginning with the same letter as :name.
  2. The second line is Directions:
  3. The third line is In a bowl, mix:
  4. The following lines should have random ingredients, where each ingredient is taken from a random file matching the corresponding letter in the :name.
  5. The last line says Cook for n minutes and serve!, where n is the number of letters in the :name.

Exercise 3: Solution

A possible solution (ZIP)