In this part, you will write a few shell scripts that work together to download Wikipedia articles.

The overall workflow is:

  1. Read a list of article names.
  2. Convert each article name into a Wikipedia slug.
  3. Download the article source for each slug.
  4. Store each article in its own file.

For example, given the input.txt file containing the following,

Cosmo Specialty Fibers
Kichio Allen Arai

your scripts should eventually produce resulting files that has the contents of those names as

Cosmo_Specialty_Fibers.wiki
Kichio_Allen_Arai.wiki

We’ll provide you with some starter code. Run the following command in desired directory to download and unzip the files. We advise you to make a new directory named hw2 under ~/cse374 directory.

wget https://courses.cs.washington.edu/courses/cse374/26su/homeworks/hw2/wiki_starter.zip
unzip wiki_starter.zip
After you unzipped, you should get two files: input.txt and get_one_article.sh.

Task 0: Printing Error Messages

Many command-line programs print error messages to standard error (stderr) instead of standard output (stdout).

Write a script named echo_error.sh that prints its arguments to stderr. This script will later be used in other scripts you write/use in the following tasks.

Expected Behavior

./echo_error.sh "Something went wrong!"

should display

Something went wrong!

as an error message (printed to stderr, not stdout).

Requirements

  • Output the given arguments to stderr.
  • Write a short usage comment after the shebang.

Hint: You can simply redirect the output of echo to stderr.

Task 1: Slugify Article Names

A URL slug is the last part of a URL address that uniquely identifies the page. Many URLs, including the ones for Wikipedia articles, use underscores (_) instead of spaces to separate words.

Write a script named slugify.sh that reads a single article name from standard input and writes the corresponding slug to standard output.

For this assignment, slugifying only requires replacing every space with an underscore.

Expected Behavior

./slugify.sh 
Cosmo Specialty Fibers

should output

Cosmo_Specialty_Fibers

Essentially we want to use this shell script to help us slugify all the article names in the given input.txt.

Hint: Take a look at the sed command.

Task 2: Download a Single Article

Warning

For this homework, our script uses a command called jq. You don’t need to know how it works, but you do need to install it by running this line curl -s https://webinstall.dev/jq | bash. The script will not work on Calgary unless you install this command.

For this part you will be adding an error check to the provided script named get_one_article.sh. Read the comment above for expected behavior and use echo_error.sh from Task 0 to output an appropriate error message.

You do not need to understand how this script works exactly but you should read the comment to understand what it does. Think of it as a program that takes in a Wikipedia slug and prints the article’s WikiText.

For example:

./get_one_article.sh Cosmo_Specialty_Fibers

prints the WikiText for that article to standard output.

Please don’t change the curl command! You do not need to know how curl or the Wikipedia API works. Simply use this script whenever you need to download an article.

Make sure that you print an appropriate error message to stderr if the user doesn’t provide the right arguments.

Task 3: Download Multiple Articles

Now it’s time to put everything together! Write a script named download_articles.sh.

This script will read in article names and slugify them into slugs. It will get the WikiText for those articles, then save the content into files.

Specifically, it should read article names line by line. Convert each article name into a slug by using the slugify.sh script. Next, use get_one_article.sh to get the article text. Then save the output into a file named

SLUG.wiki

where SLUG is replaced by the article slug.

For example,

Cosmo_Specialty_Fibers.wiki

should contain the WikiText for the Cosmo Specialty Fibers article.

Output Directory

Your script should optionally accept an output directory OUT_DIR.

./download_articles.sh OUT_DIR

If no directory is supplied, use the current directory.

If the directory is supplied, check if it is a directory. If it is, store all files under it; otherwise use echo_error.sh to output an error message.

Expected Behavior

Suppose input.txt contains

Cosmo Specialty Fibers
Kichio Allen Arai

Running

./download_articles.sh articles < input.txt

should create

articles/
├── Cosmo_Specialty_Fibers.wiki
└── Kichio_Allen_Arai.wiki

You should reuse the scripts you wrote rather than duplicating functionality.

Now you have the WikiText! Move on to the next part to finish the homework!