Lecture notes

a— title: “Lecture Notes” extra_javascript: [“assets/mermaid-10.8.0.min.js”]

1. Week 1: Introduction to Linux
2. Week 2: More commands, streams, I/O redirection
3. Week 3: More I/O, xargs, streams, cut
4. Week 4: Version Control (git)
5. Week 5: More Version Control (git)
6. Week 6: Regular Expressions, grep -E
7. Week 7: More Regular Expression, sed
8. Week 8: Users, Groups, Permissions, Persistent shell settings
9. Week 9: Introduction to Bash Shell Scripting

Week 1: Introduction to Linux ¶

What is Linux?¶

Linux is an operating system. For example, we might have a PC that runs windows or a Macbook that runs MacOS. We have a “third genre,” which we’ll refer to as Unix/Linux.

Linux is essentially another way for you to interact with your computer.

How can you use Linux?¶

Here’s how to gain access to a Linux environment: https://courses.cs.washington.edu/courses/cse391/24wi/working_at_home/

SSH & attu¶

How can you use Linux? In the basement of CSE there are a bunch of servers (computers) that run on linux. You can use ssh (a widely used network protocol that provides a secure way to access remote servers) in order to log into these computers and run commands on them.

Attu is a linux distribution that the CS department manages for us. Essentially attu is 8 powerful computers that run linux and can be assessed using the ssh program.

Vim and Emacs¶

Learn more about using editors in this course: https://courses.cs.washington.edu/courses/cse391/23au/homework/#editors

Virtual Machine¶

You can also install a virtual machine (which is essentially a virtual computer within your computer) to help you run and use Linux. This works by putting an operating system inside your operating system in your computer.

Terminal + Basic Commands¶

The terminal is a text-based user interface for interacting with your computer. Inside the terminal is the shell, which is denoted by $. Essentially, the shell is a program that allows the user to interact with the operating system + applications.

Basic Shell Commands¶

For example, you can use man find to learn more about the find command and how it works. If you type pwd and are in your home directory, you will most likely get /home/iws/username.

Flags and Arguments¶

Flags are prepended with - and change programs’ behavior slightly

• Ex. ls -l lists the contents of the current directory in long listing format

Commands take arguments

• Ex. ls dir1 (ls is the command and dir1 is the argument). Here, we’re printing our things in dir1.

Anatomy of a command:

$ ls -al dir1

ls is the command, -al represents the flags, and dir1 represents the argument(s) / parameters

Man (manual) pages – documentation¶

Documentation for Linux is built in and can be accessed using the man (manual) command.

• Ex. man ls -> provides the documentation for ls
  • This includes the structure of the command, flags, arguments, and additional information.
• You can also search man pages with /
  • If you type /reverse you can search for the word reverse in a man page

Directory Commands¶

Relative Directories¶

Week 2: More commands, streams, I/O redirection ¶

Brief History of Unix/Linux¶

Unix was invented in 1969 by Dennis Ritchie and Ken Thompson as a product of Bell Labs.

Key ideas:

• Everything is a file (directories, files, mouse)
• Having multiple users on a device
• Hierarchical file system; “gluing” together a lot of smaller files
• Documentation

File Examination Commands¶

• cat: Opening up files in vim/emacs uses memory to store file contents, and sometimes, we’d want to take the contents and print them out to the command line. You can use the cat command which prints the contents of a file to the command line.
  • Ex. cat file.txt would print the contents of file.txt to the command line
• less: The less command provides a scrollable version of cat.
  • Fun fact: man is actually being presented through less and has some of the same navigation controls (e.g. can search for the word “fruit” in esc mode using ‘n’ and then ‘/fruit’ and can quit using q).
  • Also more is the same thing as less but with different syntax (can learn more using the man page)
• head: This prints out the first 10 lines by default, but you can pass in flags to alter that behavior
  • Ex. head file.txt prints the first 10 lines of file.txt
  • Ex. head -n 5 file.txt prints the first 5 from file.txt
• tail: This prints out the last 10 lines by default, but you can also pass in flags to alter that
  • Ex. tail -n 4 file.txt prints the last 4 lines from file.txt
• wc: This prints out the number of lines, number of words, and the number of characters. We can add flags to slightly alter this behavior
  • Ex. wc -l file.txt prints out the # of lines in file.txt

Searching and Sorting Commands¶

• grep: stands for the global regex print and is pretty much the google of the command line. grep allows you to search through files for certain patterns
  • The general structure of a grep command is grep x y z w, where we’re looking for pattern x in files y, z, and w (because grep can take in multiple files).
  • Ex. grep “berry” files.txt looks for the word “berry” in files.txt, and this is what the output would look like.
  • Grep reads from stdin if we’re not given a file as a parameter
• sort: sort file.txt prints out the sorted version of the file (doesn’t change the contents of the file)
• uniq: uniq file.txt prints out the contents of file.txt but deletes the repeated lines in a file (key idea: it looks at consecutive lines).
  • We can combine sort and uniq to remove duplicate lines from standard input using sort | uniq
• find: find searches the filesystem for a file/directory.
  • Ex. find -type f -name "*.java" finds all java files (files that end in .java).
    • -type f specifies that we’re looking for files and -name allows us to search for specific file names. You can learn more about the different flags you can use with the man pages (man find).
• cut: The cut command allows users to remove section(s) from each line of the file. Just to clarify, this doesn’t alter the contents of the file, but just sends this output to stdout.

Compiling and Running java programs¶

• javac HelloWorld.java *compiles *the contents in HelloWorld.java (can replace this with any other .java file)
• java HelloWorld *runs *HelloWorld.java
• java HelloWorld.java compiles *and *runs HelloWorld.java

Standard Streams¶

• Every process has 3 streams, which are essentially abstract locations that tell a program where to read input from and where to write output to.
  • The standard streams include standard input, standard output, and standard error.
    • Stdin (Standard input)
    • Stdout (Standard output; System.out in java is an example of standard output)
    • Stderr (Standard error)
  • All of these default to the console
    • Many functions (such as ls, cd, and grep) read from user input from the terminal and print to terminal
      • In grep, the stdout and stderr default to the console/terminal
  • The shell & operating system are responsible for where stdin (standard input) comes from and where stdout (standard output) goes to. Essentially, the program receives stdin from the shell and gives stdout to the shell.

Redirecting stdout, stdin, and stderr to and from files¶

• Output Redirection (>): command > filename
  • This executes the command and redirects its standard output to the given filename
    • grep "berry" fruits.txt > berries.txt takes all the lines that contain “berry” and outputs them to the berries.txt file instead of the console.
      • Sidenote: > overrides and >> appends. So, grep "berry" fruits.txt > berries.txt would overwrite the contents of berries.txt while grep "berry" fruits.txt >> berries.txt adds to the contents of berries.txt.
• Input Redirection (<): command < filename
  • This executes the command and reads its standard input from the contents of filename instead of from the console/terminal.
  • Let’s say Hello.java takes in two arguments (two strings that represent a first and last name) and prints out Hello firstName lastName. For example java Hello "Kirupa" "Gunaseelan" (after compilation) would print out Hello Kirupa Gunaseelan. Let’s say we have strings “Matt” and “Wang” in twoStrings.txt. We could run java Hello < twoStrings.txt (after we compile Hello.java of course) which would print out Hello Matt Wang.
• Stderr redirection: 2> redirects stderr to file
  • If we had an error in our program (while running), we could redirect that and store it in a file using this command (works similarly to > but for stderr).
  • 2>&1executes the command redirects standard error to standard output
  • command > filename 2>&1 executes command, redirects standard error to standard output and redirects standard output to filename.

Pipes¶

• command1 | command2
  • This executes command1 and sends its standard output as standard input to command2
  • This is shorthand for the following sequence of commands
    • command1 > filename
    • filename <command2
    • rm filename

Week 3: More I/O, xargs, streams, cut ¶

Some more command line operators¶

• And Operator
  • command1 && command2
    • The double ampersand is an and operator that executes command1, and if that command runs successfully, executes command2.
    • If the first command fails (e.g. javac CompilerErrors.java, where CompilerErrors.java doesn’t compile successfully), the command doesn’t reach the second command. This is essentially like short-circuiting.
• Or Operator
  • command1 || command2
    • The double pipe is an or operator that executes command1, and if that command runs successfully, ends the command and doesn’t execute command2.
    • Alternatively, if the first command fails, the command moves onto the second command and executes it.
• Semicolon Operator
  • command1 ; command2
    • The semicolon operator executes both commands, regardless of whether or not either of the commands runs successfully.

Converting stdin to command arguments with xargs¶

• xargs is helpful in converting standard input into arguments that commands that don’t take in stdin can accept.
• An example of this is the command ls *.java | xargs javac or after running the command ls *.java > toCompile.txt you can run xargs javac < toCompile.txt to do the same thing. In this case, xargs takes the names of the java files as stdin and converts them into arguments for the javac command.
• find
  • The reason that xargs is better than just using *.java in the arguments section of the command is because xargs allows you to get input from more than just the current directory using the find command.
  • A command like find -name "*.java"searches the current directory and all subdirectories for java files.
  • When we pair that with xargs, we get something like this: find -name "*.java" | xargs javac to compile all the java files in the current directory and the ones below.
• Command substitution
  • One thing that you may run into on the internet as another way to use find with a command without xargs is by using a command substitution.
  • This looks like javac $(find -name "*.java") which does the same as the example above that used xargs.
  • The way this works is that the command within the $( ) is executed first and that output can be used as an argument for other commands such as javac or rm as well.

Using tee to send stdout and stderr to files and console¶

• The command tee is useful because it lets you send stdout to a file and to the console at the same time which is helpful when you want to see the output on the shell in real time, but also log what happens for future reference.
• The way to do this is to type a command like java Mystery.java | tee tee_out.txt, which runs Mystery.java and sends the output to the shell and to the file “tee_out.txt”.
• Note that this only sends stdout to the file and outputs stderr before stdout on the console, if you want to send both to the file use “2>&1”. The command should end up looking like this example: java Mystery.java 2>&1 | tee tee_out.txt.

echo¶

• echo is essentially the command line version of print.
• A command like echo "Hello World" just prints “Hello World” to the console.

Simplifying complex strings with cut¶

• The cut command can be used two primary ways, with a -c (character) flag and with a -d (delimiter) flag.
  • An example of cut -c is the command echo "abcdef" | cut -c2 which outputs “b” or echo "abcdef" | cut -c2-5 which outputs “bcde” or echo "abcdef" | cut -c2,1,4 which outputs “bad”. Those three ways to cut characters exist with the cut -c command.
  • The way to use the -d flag is to enter a character after the -d and then -f[number] with the number being the field you want to get. For example, the command echo "a,b,c,d,e,f" | cut -d, -f1 outputs “a”.
• Another way to use cut is to use it with the tail command on log files to get more specific isolated information (e.g. tail -f common_log | cut -a\" -f4 to get uw cse websites being visited).
• stdbuf
  • You can also use stdbuf with tail and cut to get more instantaneous updating with less of a buffer.
  • All that would be added to the last command to make it more instantaneous is stdbuf -oL to make this command: tail -f common_log | stdbuf -oL cut -a\" -f4.

Week 4: Version Control (git) ¶

Intro to git + terminology¶

Version Control - software that keeps track of changes to a set of files

• Ex. Google Docs where you can undo (Ctrl+Z) or look at revision history

Repository (rep) - a location that stores a copy of all the files needed for a project/assignment

• What should be inside a repo?
  • Source code files (.c files, .java files), build files (Makefiles), images, resources
• What shouldn’t be inside a repo?
  • Object files (i.e. .class files or .o files), executables

Git - a “distributed” version control system where everyone has a complete version of the repository

• There is a remote repository, which is the “central” repository
• Remote repos are hosted on services like Github or Gitlab (companies that host a git server)
• Everyone has a local copy of their repository, which is what we use to commit.
• In a basic sense, developers can push their own code + changes to other developers, and pull code from other developers.

The Four Phases in Git¶

1. Working Directory (Working changes, what’s on your computer)
   1. You can move these changes/files to the 2^nd phase (Staging Area) by staging your files using git add or git stage. This is essentially getting your changes ready, prepared, and in draft mode (preparing them for a commit later on). This is relatively easy to undo (git restore –staged <file>).
2. Staging Area/Index (change’s you’re preparing to commit)
   2. You can move your stages files to your local repo, by committing them using git commit. This saves your changes to your local repo, and is more difficult to undo.
3. Local Repository (a local copy of the repo with your committed changes)
   3. You can move your changes to the 4^th phase (remote repository) by pushing them, using git push. This is the hardest to reverse.
4. Remote Repository (remote shared repository)

Basic commands in git¶

• git clone <url> [dir]: cloning a repository from remote to make a local copy
• git add <file> …: Adds file contents to the staging area
  • Same thing as git stage <file>
  • Can be used on multiple files (accepts multiple arguments).
  • git add . stages all files in the current directory.
• git commit <file> … -m “Insert message here”: takes a snapshot of the staging area and creates a commit
• git status: view status of files in the working directory and staging area.
• git push: push changes from your local repository to the remote repository
• git diff: shows the differences between what’s in your working directory and staging area
• git log: shows you a history of commits chronologically in reverse order.
  • Consists of a commit hash, author, date, and message.
    • Commit hash - unique identifier for a commit

Git Branching and Merging¶

• Commit history (each commit builds on top of the previous one)
  • Referring to the representation below, “A” is the first commit and “C” is the most recent commit (arrows point back in time). The commits build on top of each other. For example, commit C is commit B + some extra changes.
  • HEAD - is a pointer to the git branch you’re currently working on
  • Main - the main branch of your repository (single source of truth) and the only branch created by default.
    • Main (or any branch for that matter) points to a commit which indicates where the local repo is at.
    • We can “branch” off from main, creating new branches to work on something different and then push to main (when our code is perfect).

gitGraph
    commit id: "A"
    commit id: "B"
    commit id: "C" tag: "HEAD"

More git commands¶

• git branch feature: creates a new branch called feature. You can replace feature with the appropriate name for your branch.
• git checkout <branch>: allows you to “checkout” or enter into a different branch.
• git merge <other branch>: merges the “other branch” with your current branch, updating your current branch.
  • We can merge code from one branch to another using git merge <other branch> while we’re in our current branch. This allows for our current branch to be updated with our other branch.
  • Let’s say we have changes in our feature branch (a branch named feature) that differ from changing in the main branch. First, we can merge feature into main by checking out the main branch (git checkout main) and then merging (git merge feature). If we have changes that are inconsistent within a file, then we will run into a merge conflict.
    • We can open the problematic file (using vim or emacs) and then change our file accordingly to resolve the conflicts
    • We would then have to add the file(s) and commit them to fully resolve the merge.
• git blame: shows what revision and author last modified each line of a file.

Week 5: More Version Control (git) ¶

Working with remote¶

• Here are some starting examples of local and remote repositories:

---
title: Local
---
gitGraph
    commit id: "A"
    commit id: "origin/main -> B" tag: "HEAD"

---
title: Remote
---
gitGraph
    commit id: "A"
    commit id: "B"

• origin/main
  • Just another branch, but on remote (aka main branch on GitLab)
• Conflicts may arise when pushing to main (e.g., “error: failed to push some refs to ‘[REMOTE]’“)
  • This happens when remote commits have not been pulled and a new commit is pushed.
  • Solution #1 (git fetch)
    • Brings commits from remote without merging.
    • Then use git merge origin/main and resolve this, add and commit to fix the issue and push to update the remote repository.
• Example conflict: This fails because remote and local have two different commits so the local has diverged from remote.

---
title: Local
---
gitGraph
    commit id: "A"
    commit id: "origin/main -> B"
    commit id: "C" tag: "HEAD"

---
title: Remote
---
gitGraph
    commit id: "A"
    commit id: "B"
    commit id: "D"

• Example resolution: Uses git fetch from remote and git merge to create the M commit.

---
title: Local
---
gitGraph
    commit id: "A"
    commit id: "B"
    branch remote-fetch
    commit id: "orign/main -> D"
    checkout main
    commit id: "C" tag: "HEAD"
    merge remote-fetch id: "M"

---
title: Remote
---
gitGraph
    commit id: "A"
    commit id: "B"
    commit id: "D"

• Note: git pull is an alias for git fetch && git merge

Pull/Merge requests¶

• In GitLab, it is called a merge request while in GitHub, it is called a pull request.
• These are more frequently used than merging locally.
• Steps to make a Pull/Merge request
  1. Create a local branch and make some commits
  2. Push those commits to remote
  3. Open a pull/merge request on GitLab/GitHub
  4. Collaborate with others, leave comments, and fix conflicts
  5. Merge into main (or another branch)

Creating a branch (Part 2)¶

• git checkout -b feature
  • Creates a new branch with the name “feature” in one command
• After making changes, use git add, git commit, and then git push
• The first time you try to use git push on a new branch it might give you an error since there isn’t a remote branch to push to, you can change this by running git push –set-upstream origin feature

---
title: Local
---
gitGraph
    commit id: "A"
    commit id: "B"
    branch remote-fetch
    commit id: "D"
    checkout main
    commit id: "C"
    merge remote-fetch id: "origin/main -> M"
    branch feature
    commit id: "X" tag: "HEAD"

---
title: Remote
---
gitGraph
    commit id: "A"
    commit id: "B"
    branch remote-fetch
    commit id: "D"
    checkout main
    commit id: "C"
    merge remote-fetch id: "M"
    branch feature
    commit id: "X"
    checkout main
    commit id: "Y"

• The repository graphs above demonstrate what happens when the command git push feature is executed as it pushes the feature branch from the local repository to remote. There is also the Y commit on remote because someone else, “Evil Matt”, pushed directly to main (tsk tsk).
• After this, you can make a merge/pull request on GitLab or GitHub
  • In this request you can add a title, description, etc.
• When you push the Create merge request button, a request is created, but nothing is changed.
• In the request you can now see some things:
  • You can see the log of all the commits
  • You can also see all the changes made and make comments on code
  • If there is a merge conflict, you can fix the conflict on the website and make a commit to fix the conflict which is added to the feature branch.
  • After fixing the conflicts, you can click merge and now the feature branch has been merged into main.

---
title: Local
---
gitGraph
    commit id: "A"
    commit id: "B"
    branch remote-fetch
    commit id: "D"
    checkout main
    commit id: "C"
    merge remote-fetch id: "origin/main -> M"
    branch feature
    commit id: "X" tag: "HEAD"

---
title: Remote
---
gitGraph
    commit id: "A"
    commit id: "B"
    branch remote-fetch
    commit id: "D"
    checkout main
    commit id: "C"
    merge remote-fetch id: "M"
    branch feature
    commit id: "X"
    checkout main
    commit id: "Y"
    checkout feature
    merge main id: "M1"
    checkout main
    merge feature id: "feature -> M2" tag: "main"

• Now feature is gone and points to the M2 commit in main since we have merged and deleted the branch on GitLab/Github.

More git knowledge¶

• How to undo a commit
  • Use git revert
    • git revert HEAD reverts the most recent commit, but you can use the specific hash value for any commit.
    • This command makes a new commit that undoes all of the reverted commits changes and keeps the other commit in the history.
    • git reset --hard HEAD~1 is NOT a good idea after pushing to main since it creates a conflict.\
• Git ignore
  • This is for if you want git to ignore a file or filess (e.g., a java .class file), which means that it wont ask you to add it or commit it.
  • All you need to do is create a .gitignore file and then type the file(s) you want to ignore such as *.class files.

Week 6: Regular Expressions, grep -E ¶

Introduction to Regex¶

Regular expression (regex): Essentially a description of a pattern of text

• We will be using regular expressions in the context of grep.

Basic flags + syntax (file.txt is just a placeholder for any file)

• grep -E: E stands for extended regex, and this allows us to use regular expressions in our string.
• grep -Ei: grep is case sensitive by default (and -i allows us to search case insensitively).
• . in grep matches any single character. For example grep -E “a.” matches an “a” followed by any character.
  • If we want to match a literal period we use `grep -E “.”
• grep prints out the whole line if part of the line matches the expression it’s looking for.
  • echo “Orange cat” | grep -E “a.” would match the entire line and highlight Orange cat
• ^ refers to the beginning of the line in grep. For example grep -Ei “^k” file.txt would match all lines that begin with k (case insensitively).
• Similarly, $ refers to the end of the line. For example grep -Ei “$s” file.txt would match all lines that end with s (case insensitively).
  • If we wanted to search for a line that only contains a specific pattern (ex. Starts and ends with “cat”), we’d use grep -Ei “^cat$” file.txt
• < is used to denote the beginning of the word. A word is anything separated by an arbitrary amount of white space.
  • grep -Ei “<cat” file.txt would match words that start with cat (they don’t necessarily have to be at the start of the line). This would print out “Orange cat” to standard output.
• Similarly, \> denotes the end of a word.

Or and Repeated Characters¶

• We can use the | symbol to match words that contain either of two patterns.
  • `grep -E “a|b” file.txt matches lines in file.txt that contain either a or b.
• We can use parentheses () to group expressions together
  • `grep -E “(a|b)i” file.txt matches lines that contain an “a” or “b” followed by a “t”.
• The * symbol represents 0 or more of the thing that comes before it.
  • `grep -E “a*b” file.txt matches 0 or more a’s followed by a b.
  • Be wary: `grep -E “a” file.txt would match everything in file.txt even if they don’t contain an a (because * matches *0** or more a’s).
• Similarly, the + symbol matches 1 or more of the thing that comes before it.
• The ? symbol matches 0 or 1 of the character that comes before it.
  • grep -E “e?t” file.txt matches “et” and “t” in a file.
• `grep -E “(e|a)+t” file.txt matches eeaat eeet and aat in a file.

Regular Expressions: Character Sets and Backreferences¶

• [] represents a character set (can be thought of as a set of patterns we’re looking for)
  • For example grep -E “[abcde] file.txt is equivalent to grep -E “(a|b|c|d|e)” file.txt
• Note about special characters in a character set: Special characters (i.e. “.”, “+”, “?”, etc) are interpreted literally. So, grep -E “[.]” file.txt looks for literal periods.
• The dash character - can be used to represent intervals.
  • grep -E “[a-z]” file.txt looks for all lowercase characters. This can be combined with other character sets. For example grep -E “[a-zA-Z0-9]” file.txt looks for all lowercase characters, uppercase characters, and digits.
• We can use the {} symbol after a character to find multiple occurrences of a pattern.
  • grep -E “[0-9]{4}” file.txt matches lines with 4 digits in a row. We can use a comma to represent intervals as well. grep -E “[0-9]{,4}” file.txt matches lines that have 4 or less digits in a row, grep -E “[0-9]{4,}” file.txt matches lines with 4 or more digits, and grep -E “[0-9]{2,4}” file.txtmatches lines with 2-4 digits in a row.
• Combining earlier patterns
  • grep -E “[0-9]+” file.txt matches lines with one or more digits.
• ^ represents the beginning of a line normally, but in character sets, it represents a negation.
  • grep -E “[^0-9]” file.txt matches lines without numbers.
• Backreferences allow you to capture patterns and look for them again.
  • grep -E “(..)\1” file.txt captures the first two characters (essentially represented in 1). The 1 backreferences the original pattern (it reproduces the pattern that was captured). Thus, this command allows us to look for patterns that repeat.
    • Ex. echo “abab” | grep -E “(..)\1” file.txt would print out abab to the console because it matches the pattern.
  • grep -E “^(.).*\1$” file.txt matches lines that start and end with the same character.

Week 7: More Regular Expressions, sed ¶

Intro to sed¶

sed can be thought of as the “find and replace” of the command line.

Sed syntax (how to use sed): sed -r ‘s/REGEX/TEXT/’ <filename>

• REGEX stands for the pattern (or regular expression) that we want to match
• TEXT stands for the text that we want to replace the REGEX with
• The -r flag stands for regular expression

sed in action + the -i flag

• Suppose we had a file named best.txt that has the following content: cats are the best animal. If we were to run sed -r ‘s/cats/dogs/’ best.txt, we would output dogs are the best animal to the console.
  • However, best.txt’s contents won’t be changed (it will still contain “cats are the best animal”).
• If we pass in -i flag, we can change actual contents of a file
  • How it’s used: sed -ri <file extension> ‘s/REGEX/TEXT/’ <filename>
  • For example, if we ran sed -ri.bkp ‘s/cats/dogs/’ best.txt, best.txt would be modified, and best.txt.bkp would contain the original/previous version of the file.

sed with regular expressions¶

When to use the global variable

• sed only replaces the first “regex match” on each line
  • echo “cat cat” | sed -r ‘s/cat/dog/’ would output “cat dog” (since it only matches the first cat).
• If we want to match for multiple patterns in a line, we can use g which stands for global.
  • How it’s used: sed -r ‘s/REGEX/TEXT/g’ <filename>
  • echo “cat cat” | sed -r ‘s/cat/dog/g’ would output “dog dog”

Backreferencing

• TEXT refers to literal text (for the most part), except in the case of backreferences.
• We can capture patterns using regular expressions and refer to them later on in text (to delete or move around parts of the file).
• Let’s say we have a file, names.txt, that contains first and last names (on each line), but we only want to output the first names. For simplicity’s sake, let’s assume that each line has a sequence of characters (first name), a space, and another sequence of characters (last name).
  • The command to only output first names, would be sed -r ‘s/^(.*) (.*)$/\1/ names.txt.
    • 1 references the first set of characters that were captured (the first name).
    • We need to capture both the first name, the space, and the last name, because we are replacing the ENTIRE LINE.

Special Characters and Literal Characters

• Be wary of special characters in REGEX as they CANNOT be interpreted literally and must be escaped (i.e. periods, parenthesis, etc.). Special characters (besides backreferences) are interpreted literally in TEXT.

Week 8: Users, Groups, Permissions, Persistent shell settings ¶

Users, Groups, and Processes¶

Users on Operating Systems are different users that can use the same computer, but not at the same time. Linux was created with the idea that multiple users can be using a computer at the same.

• whoami command
  • Prints out the name of the currently logged in user.
• users command
  • Prints out all the users that exist on the computer.
• pinky command
  • Prints out more information about all the users.

Super Users

• System administrators have permissions to do essentially anything on a system and give normal users just the right permissions for them.
• This helps with limiting unintentional (and intentional) harm that might be done to the system.
• To run a command as a super user use the sudo (or super user do) command which gives you administrator privileges on that command you type after it.

Groups

Typing the groups command prints out all the groups you are a part of.

• Multiple users can be part of a group and that group can be given the same set of permissions (more on this later).

Processes

• ps command
  • Prints out the processes you are currently running.
  • A process is an application you’re running.
• ps -u [username] prints out the processes that the user on the computer with the provided username is running.
• top command
  • Shows a constantly updating list of processes running and who is running them along with other helpful information such as CPU or memory usage, time running, etc.
• Processes have a PID (Process ID) and you can kill processes (only yours) using the kill command.

Misc

• The /usr directory stands for Universal System Resources which contains the /bin or binary directory.
• Helpful tool - you can reference your last command in the shell by typing !! which means that you can type sudo !! when you want to give the command you ran earlier (that may have failed) super user permissions.

bashrc, bash_profile, and $PATH¶

.bash_profile

• It is a shell script which means that it essentially goes line by line and runs any commands within it.
• The .bash_profile specifically runs every time you login to the shell.

The bash command opens a new (non login) bash shell.

• The .bashrc file is similar to the .bash_profile file, but instead of only running on every login shell, it runs on every new shell, login, or not.

Aliases

• In .bashrc you can create an alias by typing alias [keyword]="[command]"
• Make sure to have no spaces on either side of the equal sign and quotes around the command.
• To check your aliases, you can run alias on the command line.
• When making a change to .bashrc like adding an alias, instead of reloading the shell (e.g., logging in and out again), you can type source .bashrc which runs the commands in a file line-by-line.

$PATH

• echo $PATH prints out everything in the PATH variable (which is referenced in the command with the $).
• PATH in particular has : delimiters and each item is a place where the operating system looks for commands when executing different commands (like the /bin directory which houses ls).
• To add something to your path you can say PATH=$PATH:/path/to/bin but if you want that to stay for future logins to the shell, add that command to .bashrc.

Permissions¶

We can see the permissions of each file and directory in the current directory by looking at the first column of the output from the ls -l command.

The format is d rwx rwx rwx:

• The first character is either a d if the item is a directory, or nothing if it isn’t.
• The next three groups are in the order of owner (u), group (g), and others (o).
  • Read permissions are for operations such as the cat command, which only involves reading data.
  • Write permissions are for operations like vim or emacs for making changes to the file/directory.
  • Execute permissions are for running files that can be executed such as bash shell scripts.
• Each of these groups can have up to three different permissions: read (r), write (w), and execute (x).

chmod command

• chmod u+x [filename] gives the owner execute permissions.
• chmod go-rw [filename] takes away the permission to read and write from the group and others.
• The other method is by using the NNN octal codes.
  • Each is a group(owner, group, others) and the number can be from 0-7.
  • +4 is read, +2 is write and +1 is for execution.
  • chmod 444 [filename] gives read permissions to every group.
  • chmod 600 [filename] gives read and write permissions to only the owner.

Directory permissions work differently

• Read means whether or not they can run ls on the directory.
• Write determines if you can add, delete, or modify files in the directory.
• Execute is whether someone can cd into a particular directory.

umask command

• Sets the default permissions for newly created files.
• umask 0022 gives the owner full permissions and the group and others only read and execute permissions.

Week 9: Introduction to Bash Shell Scripting ¶

Shell Scripts¶

• What’s in a shell script: A series of commands that run from top to bottom
• The shell script starts with the “shabang” which is `#!/bin/bash`. Make sure to put this at the top of your file (this tells your computer to run the program as a bash script).
• To run a shell script (like script.sh), we can type ./script.sh
• However, our file must be executable (make sure to add chmod +x to the file if it isn’t executable).
• Subshells: When we run a shell script, it actually runs in a subshell (think of it as a new shell that starts over)
  • When we cd to new directories in the subshell or create new variables, that doesn’t carry over to the “parent shell.”
  • Changes to the filesystem carry over to the “parent shell” (making new files or directories in the subshell).

• Here’s an example of a shell script:

  #!/bin/bash 
  echo "Printing wd" 
  pwd 
  echo "Printing contents"
  ls 
  

Variables and Arithmetic¶

• Can create variables using the = symbol. For example, color=”red”

  • DON’T include spaces between the symbols. color = “red”, doesn’t assign a variable.
  • Quotes can be used to group things together.
  • Variables are case sensitive (COLOR != color).

• We can reference variables as such: echo “My favorite color is $color.

  • If you use single quotes, the computer interprets the string literally. So, echo ‘My favorite color is $color’ would print My favorite color is $color. If a variable name is not found, bash doesn’t throw an error; it just prints an empty string.
• Outputs of commands can also be saved to a variable and referenced later.
  • Ex. contents=$(ls) saves the output of ls to contents.

• We can use the let command to set variables equal to expressions. There are two ways we can do this. Let’s say we have a variable called a=1.

  • let b = “$a + 3” sets b equal to a + 3.
    • Make sure to include the quotes and be consistent with spacing
  • We can also use b=$(( $a + 3 )).
    • Again, spacing also really matters here.

  Arguments¶

  • Arguments are typically entered after the file/command. If we ran ./script.sh foo bar, foo would be the first argument and bar would be the second argument.
  • Can be referenced using $0, $1, $2, etc.
  • $0 is reserved for the filename (i.e. ./script.sh)
  • $1 is the 1^st argument, $2 is the 2^nd argument, and $n is the nth argument.
  • $# gives us the number (#) of arguments provided
    • Ex. ./script.sh foo bar has 2 arguments (doesn’t count the file name)
  • $@ gives us all the arguments

  For Loops¶

  • We can use seq to iterate through a range of numbers and ls to iterate over all files in a directory. The examples are shown below. Make sure to follow the syntax (i.e. spacing, semicolon, do, and done).

  #!/bin/bash 
  for i in $(seq 1 4); do 
    echo $i
  done 
  

  #!/bin/bash 
  for file in $(ls); do 
    echo $file 
  done 
  

  If statements¶

  If statements

  • Remember to follow the syntax/spacing! Examples are listed below.
    • if [ expr ]; then
      • true/false statement must be surrounded by brackets with spaces on each side.
    • We can add else or elif (for else if).
    • Must end with fi.
  • For arithmetic comparison
    • -gt: greater than
    • -lt: less than
    • -ge: greater than or equal to
    • -le: less than or equal to
    • -eq: equals
    • -ne: not equals
  • Comparison operators (and, or, negation)
    • if [ expr1 ] && [ expr2 ]; then: and
    • if [ expr1 ] || [expr2]; then: or
    • if [ ! expr1 ]; then: negation

  More comparison operators + Examples¶

  Comparison Operator	Description
  =, !=	String operator comparison
  -z, -n	Test if a string is empty (-z) or nonempty (-n)
  -f, -d	Test if a file (-f) or a directory (-d) exists
  -r, -w, -x	Test if a file exists and is readable (-r), writeable (-w) and executable (-x)

  #!/bin/bash 
  a=1 
  b=1 
  if [ 4 -lt 6 ]; then
    echo "4 is less than 6" 
  elif [ $a -eq $b ]; then 
    echo "4 equals 6"
  else 
    echo "4 is greater than 6"
  fi 
  

  if [ $a -lt 10 ] && [ $a -gt 5 ]; then
    echo “variable a is between 5 and 10”
  fi 
  

  if [ -n "$NAME"]; then 
    echo 'Variable $NAME exists'
  fi 
  

  Exit Codes¶

  • $? Is the exit code of the previous command that was run.
  • 0 means successful anything besides 0 is unsuccessful
  • exit in shell script = 0
    • ‘exit 17’ means exit with exit code of 17