---

Discussion:

Discord
Slack

---

Topics:

• Fundamental Methods
  • Introduction to AI Agents & Signal Basics
  • Understanding Waves
  • Android/iPhone Programming
  • Using Microphones/Speakers as Sensors
  • Multi-Microphone/Speaker Applications
  • Deep Learning for Enhanced Hearing
  • Programming Models on Phones
  • IMU & GPS systems
• AI & Agentic Systems
  • Real-time Translation systems
  • Proactive Agents
  • LLMs as Judges for agentic systems
  • Augmenting LLMs with Tool Use & Actions
  • AI Scientist systems
  • Benchmarks & Challenges in Evaluating Agents
  • GUI Agents

---

Course schedule:

Date	Topic	Notes
3/31	Introduction and Signal Processing Basics	Slide 1 Additional Readings: Signal and Systems course Video on Aliasing and Nyquist Video on negative frequencies
4/2	Introduction and Signal Processing Basics (cont.)	IMU Slides GPS Slides
4/7	Android and iOS development tutorial	Sample codes - Android Doc and sample codes - iOS iOS+Swift Youtube Tutorial Android Youtube Tutorial
4/9	Understanding Waves	Slide 2
4/14	Using microphones/speakers as sensors
4/16	Using multiple microphones/speakers	BeepBeep: a high accuracy acoustic ranging system using COTS mobile devices Underwater 3D positioning on smart devices
4/21	Deep learning for Enhanced hearing
4/23	Programming models on phones
4/28	Mobile Agents	Papers: Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
4/30	Mobile Agents	Papers: Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC
5/5	Web Agents	Papers: WE BAR E N A : A REALISTIC WEB ENVIRONMENT FOR BUILDING AUTONOMOUS AGENTS WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks?
5/7	Memory Agents	Papers: MemGPT: Towards LLMs as Operating Systems A-MEM: Agentic Memory for LLM Agents
5/12	Project Brainstorming	Brainstorm ideas over the weekend!
5/14	Proactive Agents	Papers: Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance LlamaPIE: Proactive In-Ear Conversation Assistants
5/19	LLM as a judge	Papers: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena PROMETHEUS: INDUCING FINE-GRAINED EVALUATION CAPABILITY IN LANGUAGE MODELS
5/21	Augmenting LLMs with Tool use/Actions	Papers: Toolformer: Language models can teach themselves to use tools ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs Gorilla: Large Language Model Connected with Massive APIs TravelPlanner: A Benchmark for Real-World Planning with Language Agents
5/26	Memorial Day (No Class)
5/28	Software Engineering and Coding Agents	Papers: CodeMonkeys: Scaling Test-Time Compute for Software Engineering SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering RepairAgent: An Autonomous, LLM-Based Agent for Program Repair SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?
6/2	AI Scientist	Papers: The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery Automated Design of Agentic Systems Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study
6/4	Project Work Session
6/9	Project Presentations

---

Assignments:

Implement head tracking using smartphone IMUs Implementing head tracking using smartphone IMUs: Assignment 1 (Due 4/21/2025)
Implementing Yolo video model on smartphone: Assignment 2 (Due 5/9/2025)

---

Project (40% total):

• Project pitch/proposal (5%)
• Project demo (30%)
• Evaluation metrics (5%)

---

Grading:

Late policy: Assignments may be handed in up to ten days late, at a penalty of 10% of the maximum grade per day.

Homework 1 (20%)
Homework 2 (20%)
Final Project (40%)
Class participation (5%)
Class presentation of one paper (15%)