Section 15 - glob and readdir

Section 15: Glob and Readdir

Section Goals

By the end of this section, you should know how to:

Use glob and readdir to find and manipulate files in a directory

Quick Announcement:

HW4 has been released, due next Wednesday (November 20th)

Searching directories with glob

In the last section, we hard-coded file names for scores.json and dictionary.txt. However, there are many situations when we want to grab lots of files, or when we are not sure what the files are called. This is where glob comes in handy.

To use glob, we must import the glob package.

const util = require("util");
const glob = require("glob");
const globPromise = util.promisify(glob); // override to get a promise version

After this, you can use glob to retrieve arrays of files or directories. Glob takes one argument, the path of the files you want, and it returns a promise whose value is equal to the array of matching files.

let scores = await globPromise("scores/*.json");
let firstScore = await fs.readFile(scores[0], "utf8")

The * can match any non-slash character, meaning you can find files with certain formats within a directory. After this line of code, scores will contain an array of file paths to these JSON files, each of which you could plug into readFile to get the contents of the JSON.

Searching directories with readdir

readdir will do about the same thing as glob, with some minor differences. First, you will want to "promisify" it as we do with readFile.

const fs = require("fs").promises;

Glob takes as an argument a pattern for the files, using stars for uncertain parts of the path, and returns full file paths. readdir (promisified) takes the path of a directory, and returns the filenames of the files within. These file names do not include the path to the file.

let scores = await fs.readdir("scores/");
let firstScore = await fs.readFile("scores/" + scores[0], "utf8")

A Cafe Directory Structure

The cafe web service now holds its data in a directory structure to help process the results. We can process these directories in various ways.

cse154-cafe/
           app.js
           categories/
             drinks/
               drinks-icon.png
               bubble-tea/
                 info.txt
                 purchase-history.txt
               classic-coffee/
                 info.txt
                 purchase-history.txt
               ...
             foods/
               ...
           public/
             fetch-menu.js
             admin.html
             admin.js
             contact.html
             contact.js
             img/
           stock-img/
             ...

Contents (abbreviated)

List what would be returned by the lines of code below:

await fs.readdir("categories");

await globPromise("categories/*");

await fs.readdir("categories/drinks");

await globPromise("categories/drinks/*/info.txt");

await globPromise("**/*.js");

Solution

cse154-cafe/
         app.js
         categories/
           drinks/
             drinks-icon.png
             bubble-tea/
               info.txt
               purchase-history.txt
             classic-coffee/
               info.txt
               purchase-history.txt
             ...
           foods/
             ...
         public/
           fetch-menu.js
           admin.html
           admin.js
           contact.html
           contact.js
           img/
         stock-img/
           ...

Contents (abbreviated)

let categoryReaddirPaths = await fs.readdir("categories");
           ["drinks", "foods"]

let categoryGlobPaths = await globPromise("categories/*");
           ["categories/foods", "categories/drinks"]

let drinksReaddirPaths = await fs.readdir("categories/drinks");
           ["bubble-tea", "classic-coffee",
            "drinks-icon.png", "the-sippy"]

let drinksInfo =
          await globPromise("categories/drinks/*/info.txt");
           ["categories/drinks/bubble-tea/info.txt",
            "categories/drinks/classic-coffee/info.txt",
            "categories/drinks/the-sippy/info.txt"]

// ** recursively searches within the current directory and all subdirectories
          let allJS = await globPromise("**/*.js");
           ["app.js", "public/fetch-menu.js", "public/admin.js",
            "public/contact.js", "public/fetch-menu.js"]

Exercise 1: Hybrids!

In this exercise, you will use glob to display photos of... animals. You can find the starter files and images in this hybrids.zip folder.

Here is a solution to reference the expected behavior, but don't peek at the code! hybrids.zip.

Slides with more details are provided below.

Overview (1/3)

The hybrids.html page uses hybrids.js (completed for you) to make GET requests to our webservice, and display images on the page using the plain text results. When the #submit-one button is clicked, a request will be made to /animal/:animal where the value of :animal is whatever the user input into the #animal text input box.

If the user clicks on #submit-all instead, a request will be made to /animals/category/all. The response will then be used to populate the #results area with the resulting images.

First Endpoint (2/3)

Write a /animal/:animal GET endpoint that uses glob so that it will return a plain text result of all images containing the string :animal (each on their own line). Use image file paths from public/images/. Remember that the * in glob matches 0 or more of any character. The starter code promisifies glob for you with globPromise so you can easily use async/await.

Hint: you will need to strip off the public/ from the front of the path.

Second Endpoint (3/3)

Next, write a /animals/category/all endpoint that does the same thing as the first endpoint, but provides all images.

It is important to note that the relative path to the images from public/hybrids.html is different than the path from hybrids-app.js. So, for the image paths you pass back to the client, you will need to strip off the public/ from the front of the path.

It's also important to avoid extra new lines at the end of the output, since those will lead to improperly rendered images. You can use the String trim method to remove trailing whitespace.

Exercise 2: Recipe Generator

It's always fun to try new things. When it comes to cooking, sometimes it's hard to find that spark of inspiration. In this exercise, you'll use folder and file processing to create a web service that takes in a single parameter name and outputs a randomly-generated recipe idea based on the letters in the name.

The first letter of the name will correspond to the first letter of the recipe name, and the rest of the letters of the name will each correspond to a randomly-generated ingredient for that recipe. The final recipe will be output in plain text.

Go "down" to work through the slides specific to this exercise.

Exercise 2: Provided Files

Download and unzip the provided recipe-generator.zip. In this folder, there are two sub-directories, foods and ingredients. foods contains exactly 26 txt files each corresponding to a lower-cased letter of the English alphabet, listing recipe names. ingredients includes many files consisting of a letter followed by a number, each file listing a single ingredient.

There is also a starting recipe-generator.js provided which you will fill in to create one /:name endpoint.

Exercise 2: Example file in `foods`


Macaroni
Manicotti
Mantou
Marmalade
Masala
Milkshake
Minestrone Soup
Mochi
Mooncake
Muffin

foods/m.txt contents

Note that there is likely a blank line at the end of the file. Since you don't want an empty entry when splitting the foods up, we recommend using the trim method of Strings to trim the whitespace from the end.

Exercise 2: Example file in `ingredients`


1 cup rolled Oats

ingredients/o1.txt contents


1 Okra

ingredients/o2.txt contents

Note that each ingredient file contains a possible ingredient starting with the corresponding letter. To help generate a "reasonable" recipe, each ingredient option includes a unit (e.g. for the ingredient "Oreo" in ingredients/o.txt, it may be listed as "1 Oreo", but there is also an option for "1 box of Oreos" for extra oreo-ness in your recipe). Your code should not depend on this format though, it just makes the output nicer.

(hint: You will need glob to sort through these)

Exercise 2: Example Output

The following is an example (random) output for a request to
/Mowgli:


Mowgli's Muffin
Directions:
In a bowl, mix:
1 Oreo
1 gallon of Water
1 oz of Green peas
1 Lentil
1 Ice cube

Cook for 6 minutes and serve!

Example output (plain text)

Exercise 2: Generating and Outputting the Recipe

Now that we know the format of the files we have in the two directories, let's implement the recipe generator.

Implement a /:name GET endpoint that takes :name as a URL parameter. It should not matter if the name has capital letters or not. This endpoint will send a plain text response with the recipe in the format outlined on the next slide.

Exercise 2: Output Format

The first line is :name's recipeName, where the recipe name is randomly chosen from a food beginning with the same letter as :name.
The second line is Directions:
The third line is In a bowl, mix:
The following lines should have random ingredients, where each ingredient is taken from a random file matching the corresponding letter in the :name.
The last line says Cook for n minutes and serve!, where n is the number of letters in the :name.

Exercise 2: Solution

A possible solution (ZIP)