Pitch-to-MIDI Converter

Implementation Plan and Schedule

The three major components of our pitch-to-MIDI converter are: A/D audio interface, FFT algorithm, and MIDI output interface. The core of the device is the FFT algorithm, therefore that portion of the project will be implemented and tested first. The A/D interface is mostly built in to the hardware; the part that we will have to handle is getting the digital data from the A/D board into an FPGA. This portion of the project should be done second (non-intensive parts like setup and measurements can be done simultaneously with the FFT algorithm). Once we are able to reliably input an audio signal and output frequency information, the MIDI output interface will become the project focus. Of course, preliminary issues and preparation can be addressed simultaneously with both of the other two major project parts.

Tentative schedule

Weeks 1 & 2 - (already past)

Week 3 - prepare FFT algorithm, acquire and test A/D hardware

  • 4/14: Acquired XStend board

Week 4 - put FFT algorithm into FPGA, interface A/D hardware to FPGA

  • 4/17: Translated VHDL codec interface to verilog and tested
  • 4/18: Debugged and tested Verilog codec interface. Works in the simulator but not on the XStend board.
  • 4/20: Prepared FFT algorithm

Week 5 - interface A/D hardware to FPGA, prepare MIDI output interface

  • 4/24: Worked on FFT algorithm.
  • 4/25: Decided on complete specs and requirements for the project.
  • 4/27: Got the audio codec working with our verilog interface and loopback modules. Wrote memory interface and the path from codec to memory. Tested 20 to 8 bit audio conversion; still working on this part.
  • 4/28: Coded a software FFT algorithm and fixed-point C library to be used in testing our hardware FFT.

Week 6 - put MIDI output interface into microcontroller

  • 4/30: Calculated floating-point values for the look-up table
  • 5/02: After much experimentation with the FFT algorithm and fixed-point arithmetic in software, we have determined that our original design will be really hard to implement in hardware. Instead of using the DFFT algorithm for pitch-extraction, research has lead us to a better method. This pitch period estimation method was proposed in 1969 by Rabiner and Gold (see diagram). We should be able to implement this process to work in real-time on the FPGA, thus reducing both our original memory requirements and latency.
  • 5/04: Built and tested the signal peaks detector--the first part of the pitch extractor circuit. Began building pitch period estimator--the second part of the pitch extractor circuit.
  • 5/05: updated specs and requirements page. Completed presentation slides for Monday presentation

Week 7 - interface FPGA to microcontroller, interface microcontroller to 5-pin DIN socket

  • 5/08: Worked on pitch extraction algorithm. Completed Pitch Period Estimator (PPE) Verilog module. This module takes incoming pulses and determines the period by averaging new period measurement with current average period measurement. After receiving a pitch, the PPE blanks for a set time (determined by current pitch estimate) and will not accept new pulses during this time. After blanking, the pulse detection threshold exponentially backs-of (simulated with counters and shifting). This part corresponds to part 3 in the Rabiner and Gold algorithm. Made a minor change to our block diagram (seen in second presentation slides) so that the bus between the FPGA and uController is 10 bits wide instead of 8 bits wide. This modification is required to give proper resolution for note frequencies in the upper rance (over about 2 kHz).
  • 5/09: Assembled MIDI-out DIN socket and uController interface on breadboard
  • 5/10: Wrote MIDI controller code for uController.
  • 5/11: Tested MIDI controller. Determined that the Oki 8051 chip's RS-232 UART does not seem to work (not sure why, but apparently no one else has had any luck with this, either). Switched to using Atmel 8051 part, which does work. Tested, without success, the MIDI controller. Also designed layout and interface for part 4 in the Rabiner and Gold algorithm. We still need to work out the details for this part and implement it in Vergilog. For this part, we may simplify the Rabiner and Gold design (slightly).
  • 5/12: Found bugs in the MIDI controller code and fixed. MIDI output controller successfully controls external MIDI synthisizer. Only minor modifications to the MIDI controller code will be required to make it play notes calculated from period.

Week 8 - test pitch-to-MIDI converter, consider functionality extension

  • 5/15: Made progress on the final module of the pitch extraction circuit.
  • 5/16: Calculated a table of MIDI note values and their corresponding hex, frequency, and period information. Found a logarithmic approximation for the relationship between MIDI note value (0-127) and counted period. This relationship is:

    MIDI Note = 150 - 12*log2(Period Count)

    Where the Period Count is based on a counter that increments every 2^8 clock cycles (upper 10 bits of an 18-bit counter). The effective frequency of this counter is thus 24 MHz / 2^8 = 93.75 kHz. The values calculated from this function, when rounded, are very accurate for the range of MIDI notes (0-128). However, it is unclear whether it would be more efficient to use a look-up table rather than a formula to convert perdiod to MIDI note; a LUT would require many more than 128 entries because several measured pitch periods can result in the same approximate MIDI note.

  • 5/17: Finished design for pitch extraction circuit. The full design took 460% percent of the FPGA when designed. This find was very disappointing. Decided to try a software simulation of the algorithm to see where we can make simplifications to the design. Implemented a simulation in Java that produces reasonable results.
  • 5/18: Working with the Java simulation, managed to trim down several aspects of the design. Of note: we converted the datapath to 8 bits from 16 bits and trimmed down some of our counter logic (in the PPE unit). Still at about 120% board capacity.
  • 5/19: The future looks somewhat brighter after running some simulations on a variety of notes (piano sounds in wave file format). The unit should function well on frequencies over about 500Hz and should work (to some extent) for lower freqencies. The reasoning behind this is that there are fewer (detectable) harmonics at higher frequencies. We trimmed down the design to the following: PPE has no exponential backoff. There is no final calculation table; a vote is simply taken of the 6 period candidates at each cycle. This scheme appears to work okay (about as good as before) in the software simulation.

Week 9 - implement functionality extension, tie up loose ends

  • 5/22: Put everything on the FPGA and debugged. Made some minor modifications to the Verilog after simulation. Also made modifications to the MIDI controller C code for the uController so that it translates the value on port P2 to a MIDI note. Preliminary testing shows that the logic on the FPGA puts a value on the output pins and the MIDI controller grabs this value and sends out a MIDI message. Whether or not the note is correct based on the pitch is yet to be determined.
  • 5/24: Finished Pitch Extraction Circuit documentation. Finalized design. FPGA capacity: 99%. There is one more feature we'd like to add that improves performance in the software simulation, but will need to squeeze out some room.
  • 5/25: Tested the design to date. At this point the synth (controlled by our microcontroller) sounds notes almost at random. It's not so bad because it's almost certain that obvious problems still exist. We also worked in software some, trying different simple statistical techniques to clean up the output. A median calculation on a group of samples looks reasonable, but so does an average. Either way, these calculations will be limited by the amount of time they'll take.
  • 5/26: Found some problems and fixed them. Added a constant threshold value in the PPEs because it seemed to clean up the software results quite a bit. Also played a lot in the microcontroller. Tried several things: averaging different numbers of samples, taking the median of different numbers of samples, skipping some number of samples, etc. FPGA simulation was a bit troublesome (lots of Xs that we couldn't explain!) but it turned out to be just a matter of declaring the clock as "1 0" instead of "0 1". The output is improving, but it doesn't seem like there's a lot more we can do..except be optimistic.
  • 5/29: Today we fiddled a lot of little things which in the end seemed to improve the output quite a bit. We decided to use a median measurement in the microcontroller for smoothing. It's still not perfect, but over a range of about 3 octaves our device can track pitch quite well (the max range might be between about 100 and 2000 Hz). It's not a matter of getting things wrong really, but outputting extra, superfluous values. We tried lots of little tweaks, but ultimately I think that the only way to really make this work right is to add complexity in the FPGA which we can't do because we're already at 100% usage. Still, it does pretty well for cutting the Rabiner & Gold algorithm down to one fifth of what it would be in it's "full glory."

Week 10 - pitch-to-MIDI converter and all documentation completed

  • 5/30: Created presentation slides for design review of Erran and Art's Pac-Man Game. The design review is in class tomorrow

    • CSE 477 Spring 2000