Lecture: OS organization

class structure

• staff
• relations to prereq
  • 333: how to write applications that use the operating system
  • this course: how to write an operating system
• goal: understand & learn to work with low-level systems software
  • navigating an unfamiliar codebase
  • reading tech manuals & finding information online
  • debugging: printf, thinking (really) hard
  • communication
• structure
  • capstone-y, programming-intensive
    • labs: based on xv6, a Unix-style teaching OS on RISC-V
    • project: form groups of 4-6, do something interesting with xv6
    • no exams
  • lectures: review basic concepts & discuss more recent ideas (virtual)
  • sections & office hours: help you with labs (virtual & in-person)
  • see the course website on grading & policies
• how to take this course
  • attend lectures if you want to participate in discussion / hang out
  • attend sections if you have questions about labs
  • feel free to skip if you’re doing well in labs/project
  • self-learning (e.g., through manuals & online) & collaboration
  • think how you’d work on a software project in the real world
• how not to take this course
  • optimizing for exams or GPA
  • focusing on theories
  • relying on class for technical topics
  • starting a lab the night before the due date

overview

• what’s an OS / why do we need an OS
  • a reusable library
  • a set of programming abstractions
  • a mediator between applications/hardware
  • concerns: safety, performance, scalability
• the computer
  • examples: IBM T42, Raspberry Pi, HiFive Unleashed
  • abstract machine model: CPU, Memory, and I/O
    • CPU interprets instructions: IP (Instruction Pointer) to memory
    • memory stores instructions and data
    • I/O to external world: e.g., memory-mapped I/O (MMIO)
  • in this class we use QEMU to emulate a RISC-V computer
• how does printing “hello” work on this computer?
  • example: a tiny hello “kernel” (tar and source code)
  • run make qemu
    • UART (serial port) at physical memory address 0x1000_0000
    • perhaps the lowest-level “hello world”
  • talk to devices through memory address space
  • • example: virt
    • example: chapter 4 (memory map) of the SiFive U54 manual
    • will see more examples in later labs
• a brief history
  • Kirk McKusick - A Narrative History of BSD (play 2:22–9:50)
  • Unix history diagram
  • what’s the OS on your phone/laptop/…?

programming interface

• recap: print to screen via MMIO (0x1000_0000)
  • not specific to RISC-V: see “hello” in x86 and physical memory map
  • discussion: why not do this directly in applications instad of using the OS? pros & cons?
    • pros of “direct” printing: better performance (and better determinism)
    • cons of “direct” printing: lack of portability, isolation (e.g., interleaved output), and flexibility (e.g., printing to different devices), abstraction
• Unix system calls & files
  • xv6: see user/user.h for system calls
  • today we will use Linux as an example, via strace

example: hello

$ strace -o log -- echo hello

output:

execve("/usr/bin/echo", ["echo", "hello"], [/* 43 vars */]) = 0
...
write(1, "hello\n", 6)                  = 6
...
exit_group(0)                           = ?

• system calls: execve (first), write, exit_group (last)
• what’s the first argument of write
• what’s the first argument of exit_group

example: uptime

$ strace -o log -- uptime

output:

execve("/usr/bin/uptime", ["uptime"], [/* 43 vars */]) = 0
...
openat(AT_FDCWD, "/proc/uptime", O_RDONLY) = 3
lseek(3, 0, SEEK_SET)                   = 0
read(3, "8827716.99 421503180.64\n", 8191) = 24
...
write(1, " 15:37:50 up 102 days,  4:08,  8"..., 72) = 72
...
exit_group(0)                                          = ?

• system calls: open/openat, lseek, read, …
• file: /proc/uptime - is this a real file on disk? (source code)
• you’ll implement a simpler version in lab fs
• other pseudo files
  • cat /proc/cpuinfo; see more about procfs
  • cat /dev/random

exercise: redirection

what system calls are used to implement redirection > (i.e., how does the output of uptime go to an on-disk file instead of terminal)?

$ uptime > file

hints:

• try strace -f -o log -- sh -c "uptime > file".
• how many processes are created? look for clone(...) (old days: fork).
• look for execve("/usr/bin/uptime", ...) and the system calls right above (openat, dup2, close).
• what do these system calls do? search online or use manpages.

exercise: pipe

what system calls are used to implement pipe |?

$ uptime | tr '[a-z]' '[A-Z]'

hints:

• try strace -f -o log -- sh -c "uptime | tr '[a-z]' '[A-Z]'".
• look for calls to pipe, clone, execve, and dup2.
• how many processes are created? what are their stdin/stdout (file descriptors 0 & 1)?

summary

• system calls (and “files”)
  • why do we need them at all (compared to just library functions)? isolation & protection
  • what is the “file” abstraction
    • as long as you can open/read/write/close
    • can support different “backends”: terminals, disk files, pipes, sockets
• guess how strace is implemented in Linux
  • search for “ptrace”
  • what if we want to implement it for xv6?

isolation

• lab util question:
  • why do we have to call exit in main? what if we simply return without calling exit?
  • try to figure out what happens, why, and how to fix this. we will revisit next week.
  • “meta” skill: how to get unstuck?
• what have we seen so far
  • memory-mapped I/O (e.g., printing)
  • abstractions (e.g., files instead of raw devices)

    +------+    +------+     +------+
    | app1 |    | app2 |     | app3 |
    +------+    +------+     +------+
    +-------------------------------+
    |       operating system        |
    +-------------------------------+
    +-----+  +-----+  +-------------+
    | CPU |  | RAM |  | I/O devices |
    +-----+  +-----+  +-------------+
    

• goal: isolation & sharing
  • a process should not be able to
    • corrupt the memory of the kernel or another process
    • consume all the CPU time
  • while still being able to communicate with other processes & hardware
    • otherwise, just use a separate machine for each application (why not?)
• real-world example: how do we manage a large building like Gates
  • partitioning (who can access which rooms)
  • virtualization (“Bezos Seminar” for Gates G04)
• memory
  • keys
    • split memory into chunks and assign a “key” to each chunk
    • each process holds one or more keys
    • check (who? when?) if the process has the key to access the memory chunk
    • examples: RISC-V’s physical memory protection (PMP); see §3.6 of priv spec or kernel/start.c
  • virtual memory
    • suppose process A writes some value to a memory address
      • can process B read the value from the same address?
      • can process B overwrite the value in A? why not?
    • naming issue
      • hide physical memory addresses from programs
      • name memory using: <address space, virtual address> instead of <physical address>
    • isolate two processes
      • need some virtual-to-physical address translation (by what?)
      • use a safe language: e.g., write process in Java (any other example?)
      • virtual machine: e.g., can xv6 in QEMU harm your laptop?
      • hardware: MMU - segfaults if violated
    • isolate kernel memory from processes
      • take a look at kernel/memlayout.h in xv6
      • can a process write to kernel? who checks?
• time
  • cooperative scheduling
    • each process voluntarily gives back the control (to whom?)
    • what if they don’t
  • virtual CPUs
    • each process thinks it has a dedicated CPU
    • kernel runs processes in turn on physical CPUs: save/store state
      • example: take control back from a process that spins forever
      • timer interrupt & preempt
• many plans to achieve isolation
  • this class focues on isolation through kernel/user split
  • will talk about other techniques in the second half of this quarter (think how Java/QEMU work)

kernel-user split

• CPU support: kernel/user mode flag (ring)
  • RISC-V privilege modes: see priv spec, chapter 1
    • M (machine), S (supervisor), U (user)
    • x86/arm have similar modes
  • kernel run in S-mode & can execute privileged instructions
    • examples: change the address translation map
    • including changing the current privilege mode
  • user processes run in U-mode and are unprivileged
• how do unprivileged processes work
  • they cannot directly access files, network, etc.
  • the kernel can
  • can they simply jump into an kernel address? no - what prevents them from doing so?
  • system calls: controlled transfer
    • user → kernel: ecall instruction
    • kernel → user: sret instruction
• what to put below/above the system call interface: a comparison
  • monolithic kernel
    • big kernel (including file systems, network, etc.)
    • easy to develop applications
    • hard to isolate kernel components
  • microkernel
    • kernel + user-space servers (e.g., file system, network)
    • applications talk to servers via IPCs
    • performance issues: IPCs may be slow
  • most real-world kernels are mixed: Linux, macOS, Windows
  • debate: example