Lecture: System calls, exceptions, and interrupts

preparation

• read the xv6 book: §4, Traps and system calls and §5, Interrupts and device drivers

overview

• review function calls
• trap handling: exceptions, system calls, interrupts
• lab alarm: be careful with pointers and address spaces; for example, don’t access (load/store/execute) a user-space pointer in the kernel directly, as the value of pointer value has no meaning in the kernel address space.

function calls

• review CSE 351: calling conventions, register saving conventions
• specification
  • RISC-V psABI
  • RISC-V base spec: §26, RISC-V Assembly Programmer’s Handbook
• integer registers: x0-x31
  • x0/zero
  • x1/ra (return address)
  • x2/sp (stack pointer)
  • x8/s0/fp (frame pointer)
  • x10-x17/a0-a7: argument registers; return value in a0
  • which registers are preserved across calls?
• instruction pairs
  • auipc rd, imm20 (add upper immediate to pc): add imm20 << 12 to the address of this instruction and write the result in register rd
  • jalr rd, imm12(rs) (jump and link register): set pc to rs + imm12 (sign-extended, LSB cleared); write pc + 4 to register rd
• pseudoinstructions
  • jalr rs: pseudoinstruction for jalr ra, 0(rs)
  • ret: set pc to ra, pseudoinstruction for jalr zero, 0(ra)
  • call imm32 (usually with gcc -S): PC-relative call with 32-bit offset, pseudoinstruction for auipc ra, imm32_hi; jalr ra, imm32_lo(ra)
  • see more in tables 26.2 and 26.3, RISC-V base spec
• example

long bar(long x)
{
  return x;
}

long foo(long x)
{
  return bar(x) + 1;
}

long main(void)
{
  return foo(0) + foo(1);
}

$ riscv64-unknown-elf-gcc -O -fno-omit-frame-pointer -mcmodel=medany -mno-relax -fno-pie -no-pie -fno-inline -o foo foo.c
$ riscv64-unknown-elf-objdump -d foo
...
0000000000010188 <bar>:
   10188:       1141                    addi    sp,sp,-16
   1018a:       e422                    sd      s0,8(sp)
   1018c:       0800                    addi    s0,sp,16
   1018e:       6422                    ld      s0,8(sp)
   10190:       0141                    addi    sp,sp,16
   10192:       8082                    ret

0000000000010194 <foo>:
   10194:       1141                    addi    sp,sp,-16
   10196:       e406                    sd      ra,8(sp)
   10198:       e022                    sd      s0,0(sp)
   1019a:       0800                    addi    s0,sp,16
   1019c:       00000097                auipc   ra,0x0
   101a0:       fec080e7                jalr    -20(ra) # 10188 <bar>
   101a4:       0505                    addi    a0,a0,1
   101a6:       60a2                    ld      ra,8(sp)
   101a8:       6402                    ld      s0,0(sp)
   101aa:       0141                    addi    sp,sp,16
   101ac:       8082                    ret

00000000000101ae &lt;main>:
   101ae:       1101                    addi    sp,sp,-32
   101b0:       ec06                    sd      ra,24(sp)
   101b2:       e822                    sd      s0,16(sp)
   101b4:       e426                    sd      s1,8(sp)
   101b6:       1000                    addi    s0,sp,32
   101b8:       4501                    li      a0,0
   101ba:       00000097                auipc   ra,0x0
   101be:       fda080e7                jalr    -38(ra) # 10194 <foo>
   101c2:       84aa                    mv      s1,a0
   101c4:       4505                    li      a0,1
   101c6:       00000097                auipc   ra,0x0
   101ca:       fce080e7                jalr    -50(ra) # 10194 <foo>
   101ce:       9526                    add     a0,a0,s1
   101d0:       60e2                    ld      ra,24(sp)
   101d2:       6442                    ld      s0,16(sp)
   101d4:       64a2                    ld      s1,8(sp)
   101d6:       6105                    addi    sp,sp,32
   101d8:       8082                    ret
...

• stack

             .
             .
             .
+-> +----------------+   |
|   | main's ra      |   |
|   | previous s0/fp ----+
|   | saved  s1      |
|   |       ...      |
|   +----------------+ <-+
|   | foo's ra @main |   |
+---- previous s0/fp |   |
    +----------------+   |
             .
             .
             .

• recall backtrace from the page faults lecture
  • how to print the stack trace (e.g., during a kernel panic): check backtrace() (kernel/printf.c)
  • stack frames linked by frame pointers (s0/fp)

traps

• review CSE 351, system control flow
  • “normal” control flow - OS is no involved
  • “exceptional” control flow
    • synchronous exceptions (exceptions / syscalls)
    • asynchronous exceptions (interrupts)
  • once applications start, when do we need the OS kernel again?
    • handle errors or on-demand requests (e.g., growing stack as in lab lazy)
    • provide services (e.g., applications invoke system calls)
    • handle devices (e.g., a key is pressed, network packets arrive, time is up)
• trap handling workflow on RISC-V
  • kernel writes the address of trap-handling code to stvec (supervisor trap vector base address register); search for “w_stvec” in kernel/trap.c
  • upon exceptions/syscalls/interrupts, the cpu:
    • saves the current program counter in sepc
    • switches to kernel mode if currently in user mode
    • jumps to the address specified in stvec
  • upon sret instruction: the cpu returns to the address specified in sepc and switches to the orignal (user/kernel) mode

exceptions

• demo: add printf("%d\n", *(volatile char *)0xdeadbeef); to main() in user/echo.c
  • scause: 13 (0xd), “load access fault”
  • sepc: pc of the faulted instruction
  • stval: the virtual memory address

$ echo
usertrap(): unexpected scause 0x000000000000000d (load page fault) pid=3
            sepc=0x000000000000001a stval=0x00000000deadbeef

• exception path: user to kernel
  • window 1: make qemu-gdb
  • window 2: make gdb
  • window 2: type b *0x1a (lbu instruction at the address from sepc)
  • window 2: press c a few times to continue until window 1 runs to shell
  • window 1: run echo in the xv6 shell
  • window 2: hit breakpoint 1: lbu instruction at address 0x1a
  • window 2: use layout split to display both assembly and C source (if available)
  • window 2: single step in gdb using si
  • window 2: now in trampoline code uservec (kernel/trampoline.S) at address 0x3ffffff000
    • where is this value from? hint: TRAMPOLINE defined in kernel/memlayout.h
    • is the page table of the current process or that of the kernel is in use?
  • window 2: single step until to C code (usertrap() in kernel/trap.c)
  • window 2: what does p->trapframe->epc = r_sepc() do? save the user program counter (i.e., 0x1a)
  • window 2: exit() kills the current process
• kernel to user: what if we comment out p->killed = 1 (usertrap() in kernel/trap.c)?
  • the program traps repeatedly instead of being killed - why? use gdb to find out
  • the execution continues to usertrapret()
  • what does w_sepc(p->trapframe->epc) do? write the saved user program counter (i.e., 0x1a) to sepc, where cpu will resume execution upon sret
  • what does ((void (*)(uint64,uint64))fn)(TRAPFRAME, satp) do? jump to trampoline code userret (kernel/trampoline.S)
  • single step using si until sret, which returns to user: the same lbu instruction at address 0x1a
• q: how to change the kernel to make the process continue execution after the trap?
  • hint: add p->trapframe->epc += 4
  • it prints out some garbage value
  • how to change the kernel to make the process print out 42? (hint: add p->trapframe->a1 = 42)
• q: trampoline code is mapped in both user and kernel page tables. at what virtual address? why?
• exception path within the kernel is similar: try to introduce a page fault in the kernel and single-step using gdb yourself

System calls

• review CSE 351, system calls in x86-64

#include <sys/syscall.h>
#include <unistd.h>
/*
 * syscall number & return value: %rax
 * syscall arguments: %rdi, %rsi, %rdx, %r10, %r8, %r9
 * %rcx and %r11 are not preserved
 *
 * To compile, use `gcc -nostdlib`.
 */
void _start(void)
{
        int fd = 1;
        char buf[] = "hello world!\n";
        size_t n = sizeof(buf) - 1;

        asm volatile ("syscall"
                : /* ignore output */
                : "a"(__NR_write), "D"(fd), "S"(buf), "d"(n)
                : "rcx", "r11", "memory"
        );
        asm volatile ("syscall"
                : /* no output */
                : "a"(__NR_exit), "D"(0)
        );
}

• system calls in RISC-V
  • ecall instruction - see the priv spec
  • calling convention
    • syscall number: a7
    • arguments: a0–a5
    • return value: a0
  • example: xv6
    • usertrap() (kernel/trap.c): calls syscall() if scause is 8 (i.e., ecall from user space)
    • syscall() (kernel/syscall.c): dispatches based on a7 & sets a0 to return value
• example: write(fd, buf, n) in xv6
  • sys_write (kernel/sysfile.c): how to retrieve the values of fd, buf, and n? from what registers? where & when are these registers stored?
  • how to access the content of buf in kernel?
    • buf is a user-space pointer; we cannot simply dereference it in kernel
    • needs to copy across address spaces; it boils down to copyin() (kernel/vm.c)

Q&A

• recap
  • trap handling: user-kernel switch/communication
    • how can kernel take back control: exceptions
    • how can user request help from kernel: syscalls
    • how can kernel notify user: device interrupts - more on this next week
  • CPU support
    • registers: stvec, sepc, scause, stval, sstatus, sscratch
    • instructions: ecall, sret
  • OS examples
    • xv6: kernel/trampoline.S, kernel/trap.c
    • Linux: kernel/entry.S (handle_exception, handle_syscall), kernel/syscall_table.c (sys_call_table)
  • virtual memory layout
    • compare xv6 and Linux
    • what’s the implication of mapping kernel to higher half and leaving lower half to user space? what about copying syscall arguments?
  • hot path; must be very careful with security
    • no direct pointer dereferences, no information leakage through registers
    • time-of-check to time-of-use
    • examples
      • Intel sysret privilege escalation
      • Meltdown
• what’s the cost of a syscall? mode switch, register save & restore, page-table switch, argument copying
• how to reduce the syscall cost?
  • scatter/gather I/O: writev
  • downloading code into kernel: eBPF
  • asynchronous I/O: io_uring
  • don’t switch to kernel: vDSO, library OS/unikernel
  • exercise: speed up uptime in xv6, by mapping a read-only page to eliminate the need for kernel crossings