Assignment 2 (Due 5/7/2025)
The goal of this assignment is to familiarize yourself with the process of deploying a model onto a mobile device.
You will learn to deploy YOLOv8 onto a mobile device, and also fine-tune it on some custom data.
Part 1: Create an app that can view camera data
1) After creating the application, use this
ViewController and
SceneDelegate.
Android users please try following along with this
link, and feel free to use any tutorial you find online.
2) Build and load onto your device, you should see a square live view of the camera.
Part 2: Train YOLO
1) To train YOLO, please copy this
colab notebook onto your own Drive.
If you're an Android user, please use this
colab notebook.
2) Go through the notebook until you have downloaded the mlpackage zip archive, or tflite model, respectively.
Unarchive this directory, and make sure to add the .mlpackage extension (for iOS).
Drag and drop the mlpackage into xcode into the same directory as your ViewController.
Android users may follow with these
steps, again feel free to use any online resources for help.
Part 3: Run model on phone
1) Replace your ViewController with
this.
2) Inside setupYOLOModel, replace the empty string of forResource with the name of the model you exported from colab.
3) Fill in the TODO of captureOutput. This is responsible for converting the image into a format that YOLO can understand.
4) Fill in the TODO of detectObjects. Here, you will be creating an MLRequest which passes your pixel buffer into the model.
5) Fill in the TODO of processDetectionResults. The output of YOLO will provide many results, filter the results using an appropriate confidence threshold.
Android users please follow along
here.
Demo
Show a screen capture of your YOLO model running real time on your device!
Extra Credit
Fine tune the YOLO model on a custom dataset you can find online (or make your own!). Show a demo of your fine-tuned YOLO running on your phone.
Deliverables:
- Upload the demo in Canvas.
- Short Answer: How did the behavior of the model change depending on the confidence threshold?
- Short Answer: Please provide any feedback for this assignment so it can be improved upon next quarter!