Last month, OpenAI released the next evolution of its audio model. It is, in my opinion, much more exciting than its previous thinker-speaker models because it is full-duplex (FD).
Full-duplex models have interested me since Moshi. Not because the world needed more entities interrupting each other, but because they move us away from the turn-taking paradigm. This is one of the core problems we must solve on the way to agents interacting effectively in meatspace. If your robot had to stop acting every time it saw or heard something, it would be pretty shite.
In this paradigm, we no longer throw one large prefill block into a language model and wait for it to finish generating before it can listen again. Instead, the model advances on a regular clock. At each step, it takes in the latest sensory input and produces the next slice of output, which may be speech or silence. It can therefore hear and speak simultaneously.
In this series of blog posts I will attempt to distil a model capable of running comfortably on my computer. I will likely be nerd sniped many times.
Review:
- how ears work
- how microphones work
- FD speech models
- NACs (neural audio codecs)
- distillation techniques
Then we define our evaluation suite measuring linguistic ability, acoustic quality, and duplex behaviour. Finally, we distil existing full-duplex models under a fixed compute budget, reduce their size until they fail, and work out which capabilities resist compression. Good? Good.