The Human Layer with Laurent Mazare, The Voice of AI #Gradium

TL;DR

• Laurent Mazare, former CTO of Kyutai and now at Gradium, bridges cryptography, quantitative finance, and voice AI research with a foundation in theoretical computer science and functional programming.

• He co-founded Kyutai, a non-profit frontier AI research lab that announced Moshi, the first open source full duplex speech-to-speech model.

• Gradium emerged as a commercial spin-off to bridge research and products, focusing on voice AI that optimizes latency, expressivity, and controllability for business applications.

• His technical approach centers on Depth Transformer architecture, using autoregressive models with audio tokenizers to achieve real-time inference on consumer hardware while maintaining quality.

• Laurent views code as context-dependent: flexibility for research, robustness for production, with modern type systems providing runtime guarantees that enable fearless refactoring.

• He dismisses AGI threshold debates, focusing instead on practical model improvements, and identifies reliability and training efficiency as AI's critical missing pieces.

• His philosophy prioritizes understanding fundamentals deeply over chasing trends, enabling rapid adaptation as the field evolves, with research and engineering converging toward building the best possible technology.

From Finance to Voice AI

Since this is a portrait and some people might not know you or your work yet, could you start by introducing yourself?

My name is Laurent Mazare. I come primarily from a theoretical background. I did my PhD at the intersection of cryptography and formal methods, but I have also been deeply involved in computer programming for as long as I can remember, starting with Pascal on a Macintosh Classic. I spent a little over ten years in London working in quantitative finance. More recently, I helped start Kyutai, a non profit frontier AI research lab, where I served as CTO. Over the past few months, I have been involved in building Gradium, a spin off from Kyutai that aims to develop voice AI products based on the research we have conducted over the last few years.

Can you explain the genesis of Kyutai, the open-source lab you co-founded in late 2023?

I started getting involved with what would become Gradium in May 2023. I was approached by a former colleague from Google DeepMind about launching a non profit research initiative and securing significant funding to enable large scale research. We spent the summer brainstorming around this idea, but at the time I would not have bet on it actually happening, as there were many uncertainties surrounding the project. Things accelerated in the fall of 2023, and we ultimately announced the lab in November.

The first major research project we undertook focused on voice AI. The field was at a tipping point. Research had accelerated significantly over the previous few years, systems were starting to work, but we were still far from a true scaling phase. Competition was therefore more about having the right research ideas than about access to massive amounts of compute. Neil Zeghidour and Alexandre Défossez were also part of Kyutai’s initial team and are world class experts in the domain, with very clear views on the key research challenges and how to address them. In retrospect, we were quite lucky, as most of the research directions paid off quickly, allowing us to announce Moshi, the first open source full duplex speech to speech model, in the summer of 2024.

Competition in this space is already intense. To name a few, OpenAI offers arguably one of the most impressive voice experiences with ChatGPT audio, and ElevenLabs covers many use cases with a comprehensive toolbox for teams looking to add voice to their products. How will Gradium be better and or different?

There is certainly a lot of competition in the space, but it remains a fairly young field, with ElevenLabs only starting in 2022. We believe there are still major challenges that have not yet been fully addressed to make interactions with voice agents truly seamless. Latency remains suboptimal, efficiency which directly drives the cost of interactions can be significantly improved, and the overall experience still feels less natural than speaking with a human.

At Gradium, we believe we are uniquely positioned thanks to deep expertise on the research side, with Neil and Alex having invented neural audio codecs, combined with a strong engineering culture capable of building cutting edge products on top of this research.

Since the teams have split between the two entities, was it obvious for you [Laurent] to work actively on Gradium? How is the collaboration between Kyutai and Gradium working? Will Gradium release its own models?

To me, Gradium feels like an opportunity to bridge the last mile between research and making the technology generally available to the public. It’s a great mix of research and engineering, as only a strong understanding of both sides makes it possible to build the next generation of voice agents.

Kyutai continues its mission around open research on frontier models. For example, we recently announced a partnership with General Intuition on world models. Kyutai will also continue doing cutting edge research on audio models and publishing both the methodologies and the code for these models. These models are more proof of concepts than actual products, as the goal is to crack fundamental research challenges. Gradium reuses these research ideas and extends them into full products. The models are retrained from scratch, handle more languages, use higher quality datasets, and the research ideas are pushed further to optimize for what customers care about: low latency, voice expressivity, and the ability to control the interaction.

At Kyutai, you and the team deliberately stayed small and agile, as a research group of fewer than 15 people, with, as we often joked at each release, little emphasis on marketing or design despite major technological breakthroughs. You are now part of a commercial product, facing not only research challenges but a much broader range of technical considerations, such as reliability, scale, and UX, along with high requirements across all functions, as a service and a brand competing with the best.

How do you approach this challenge? How exciting is it, both for you as a researcher and at the company and organizational level?

It’s a big shift, but also a natural follow up to what we did at Kyutai. From a research perspective, our priority was to build new technologies that could handle use cases that were not possible before. This favors small teams, fast iteration, and a strong bias toward getting to the bottom of things. What changes is that research success is no longer the finish line. Models now have to run reliably, at large scale, and with latency guarantees. They also need to integrate cleanly into products that users interact with every day. This forces a much tighter loop between research, systems engineering, and product thinking.

It’s a new kind of difficulty, but also a very exciting one, as it rewards the deep technical understanding we have accumulated over the past few years. At the organizational level, we also have to learn a lot about building the brand, selling products, and related aspects. We’ve found amazing people to help us there, and so far it has been a great experience

Deep Dive into Voice AI

Voice has made huge leaps over the past few years. What are still the main challenges in generative AI for voice?

The main challenge for voice agents is conversational flow. It’s a complete experience where all the different aspects have to work well, from turn taking detection, which has to be as low latency as possible, to generating a voice with the best possible quality. It’s not possible to deliver a good experience if any one of these aspects is lacking.

Consistency and control are also open problems. You want voices to be expressive and natural, but also steerable by the user.

I recently wrote a piece called Voice Is the Way, to shared my growing passion for voice. Even though I still prefer reading (let’s call it unilateral voice communication, I hate voice memo!), I have little doubt that usage patterns will change with UX improvement, and especially with the rise of robotics. Do you have any predictions of your own, market or usage wise, for the years to come? As Gradium is currently built, you are providing a tool. Have you ever considered becoming more than an enabler for other products?

I also believe that usage is very likely to change as the quality of interactions improves and costs decrease. On the technical side, interactions become more natural, latency decreases, and the rough edges of interacting with an AI by voice are likely to be addressed in the next few months. From an economic standpoint, current pricing levels require a single audio generation to be consumed by many listeners in order to be economically viable. This makes it prohibitive to generate audio content that is interactive and tailored to the user. With Gradium, we believe we can both improve interaction quality and significantly reduce costs at the same time, enabling new interactive use cases.

Our strengths lie in training models and optimizing inference, so we focus on providing our in house models behind an API. Building fully consumer facing tools is a different challenge.

It requires domain specific product expertise that we don’t have, as well as the ability to sell directly to end users. An extreme example would be video games. It seems clear that some games will integrate interactive audio interactions. We are very happy to collaborate on this by bringing our expertise in audio modeling, but even if a few of us are video game nerds, we are not game designers.

Last time we met, you mentioned that we had discussed the duality between autoregressive and diffusion models, and you briefly brought up flow based models. Could you walk us through the major approaches, their respective strengths and weaknesses, the choices and experiments you have made, and perhaps a few newer directions you find promising for the future, and why?

We did indeed touch on alternative approaches beyond autoregressive models. In a nutshell, these models differ in the probability distributions they end up modeling and in how generation is actually performed.

Autoregressive models have been predominant when it comes to text generation. They are the most conceptually straightforward. These models learn the distribution of the next token conditioned on previous tokens. At each step, random sampling is performed to generate the next token. Their main strengths are the simplicity of the approach, how easy they are to train, and how well they scale to large amounts of data. The main drawback is that generation is inherently sequential, which makes it hard to parallelize. In addition, the model cannot easily make high level decisions first, such as planning an answer, before refining details like the exact wording. Thinking modes are one way to work around this limitation, but they come with significant inference costs.

Diffusion models take the opposite approach. Instead of modeling a sequential process, they are trained on a gradual noising process and learn to invert it. As a result, they are very good at making long term decisions first and then refining details by progressively producing a sharper output. Their main downside is potentially high inference cost, due to the need to run multiple denoising steps, although distilled models can reduce the number of steps required at runtime. A great video explaining these models was published on the 3Blue1Brown channel, which I also highly recommend more generally.

Flow based models sit somewhere in between. They use an invertible transformation to map data to a simple base distribution, and can be seen as diffusion models with a single step.

At Kyutai, we mostly use autoregressive models for audio generation. We rely on an audio tokenizer, mimi, to convert raw audio signals into discrete tokens (Editor’s note: Discrete audio tokens are short, quantized representations of sound mapped to symbols so models can process audio like text). With mimi, we obtain 32 audio tokens per 80 ms timestep (Note of the editor: a timestep is a fixed slice of time in the audio stream).

Having a transformer generate all these tokens directly would be computationally expensive. That is why, for Moshi (paper) and our TTS models (paper), we introduced a “Depth Transformer” architecture. A large transformer runs once per timestep, and a smaller, context free transformer expands its output into the 32 audio tokens required for that timestep. This makes real time inference on consumer hardware feasible, but it still requires some compute.

For example, on an iPhone we only model 8 audio codebooks instead of 32, which results in a small drop in quality. Using flow based models allowed us to work around this limitation and generate the equivalent of all 32 audio tokens in a single step. More details can be found in the CALM paper.

Voice models are relatively small compared to LLMs, but they rely on them heavily. As a result, there are good and bad combinations, with an inherent trade off between low latency and quality. Can you explain the main challenges you face and how you approach evaluation? And how do you adapt to clients’ needs, for example when some favor speed over intelligence or knowledge?

The core challenge here is building a fully streaming system. In a cascaded setup, the input audio needs to be transcribed in real time, the result passed to an LLM, and as soon as the end of the turn is detected, the LLM should start producing text. Text to speech must then generate audio quickly enough so that the full end to end latency stays below 500 ms. Coordinating so many moving pieces is tricky, and ensuring that no single component significantly increases latency is a real challenge.

In a sense, the Moshi setup was simpler. It is a full speech to speech model, so there is only a single model to run and no coordination is required. This allows for very low latencies, for example around 200 ms end to end, and the full duplex setup even makes it possible for the model to interrupt the user. The difficulty, however, is that such audio models are hard to control, which makes them a poor fit for business use cases where operators need guarantees about agent behavior.

With Gradium, we went back to training cascaded models and optimizing for overall latency. Being able to plug in any LLM is a major advantage, especially for integration into architectures that already rely on an LLM, possibly with some form of RAG or ways to inject client knowledge into the model. We still want low latency, so the faster the time to first token (TTFT) of the text model, the better. After that, LLM throughput is generally sufficient for real time constraints not to be an issue.

There are some simple techniques to improve perceived latency. For example, a smaller and faster LLM can be used initially to generate a first sentence, before a larger LLM takes over and provides more detailed answers. In that case, the smaller model needs to be prompted carefully so that it does not produce a definitive answer that would constrain the larger model. We do not apply these techniques ourselves, as our setup is fast enough without them, but that is because we use local LLMs, up to Qwen 3 235B.

Network latency is an additional potential bottleneck. It is better for the LLM to be as close as possible to the coordination code handling speech to text, text to speech, and the LLM itself. Our experience with different LLM APIs is that most providers exhibit large variance in TTFT, ranging from good performance around 200 ms to much higher values, sometimes up to a full second. Even if this only affects the 95th percentile, it is problematic for maintaining fluid interactions, so we carefully measure the latency characteristics of different providers. With local models, it is easy to achieve an almost constant TTFT on the order of 100 ms. This is what we observed with Gemma 3 27B running on an H100 without quantization using vLLM.

One area where we are actively improving is end of turn detection, that is, the time it takes for the agent to realize that the user has finished speaking and to forward the query to the LLM. Rather than relying on a separate module such as Silero, we use our speech to text model to also inform the agent when the user is done speaking. The advantage is semantic voice activity detection. The model can use information from previously recognized words to decide on the end of a turn, and it can even be tuned so that it does not wait for the user to fully finish their sentence before sending data to the LLM. This also makes end of turn detection more robust.

Traditional approaches rely on simple energy based models that monitor frequency bands associated with speech, but these are easy to disrupt, for example when other people are speaking in the background or when there is significant background noise. We can improve on this by training our model on generated datasets so that it learns to discard parasitic noise. In a sense, this is similar to how we made Moshi robust to noise. This is illustrated in a video where Moshi was able to listen to Neil even though the noise level was high enough that Neil himself could not hear Moshi’s voice from the laptop speakers.

You impressed much of the industry last year with Hibiki, a real time speech to speech translation model running on a smartphone GPU and translating Neil’s French to English live. Can you tell us more about how it worked, and why this capability is not part of Gradium’s offering?

The model is trained purely on speech and does not rely on an intermediate text representation. It even learns the appropriate delay for translation. If the same technology were applied to German, where verbs often come at the end of sentences, the model would learn to introduce pauses so that it has enough context before continuing the translation.

We published a paper with all the technical details here.

Getting it to run on a smartphone was a fun challenge. We used MLX and Swift to get the most out of the latest iPhone hardware.

That said, Hibiki is a research project and currently only handles French to English. Tom, the main author of the research and the paper, built it during his Master’s internship and has since started a PhD focused on improving Hibiki’s capabilities. It is still a nascent technology, with a lot of research left to be done. We did not include it in Gradium’s first product, as we were very focused on speech to text and text to speech. Depending on how the research progresses, we may well integrate similar models in the future. The main challenges will likely be efficiency, robustness, and scaling to many language pairs.

Laurent’s Evolution

Coming back to you personally, when did you start thinking about pursuing this path in tech and research? What triggered it for you? Was it a book, a movie, or a piece of software?

Very early. I started programming as a child, mostly out of curiosity and for fun, trying to build small video games on a Macintosh with a black and white display. You had to make the pixels blink as fast as possible to get some kind of gray. This was long before I thought of it as a career. What really stuck with me was the idea that you could have full control over the computer if you spent enough time understanding how it actually handles your instructions. That fascination never really went away.

Even today, I still dig a lot into low level issues, and it’s great to be able to grasp the whole stack, down to the operations that actually occur on the streaming multiprocessors of a GPU.

And in your early days, what was your first legitimate piece of code, the first one you could truly be proud of? What was it, and when did it happen?

It depends a lot on what you call “legitimate”, as a kid I wrote small video games in Pascal on a mac. I also remember writing some small simulation for planet orbits. Nothing very sophisticated but that was from a time where there was no internet so you had to get books and read magazines to learn about how to make the most out of your hardware.

You spent most of your career in finance, working as a quantitative analyst or researcher on trading systems, with a brief stint at Google DeepMind in between, before co founding Kyutai and then Gradium, with a focus on voice. Do you see a silver lining in this trajectory?

Something common to all these projects is that they require a deep understanding of the technology, as well as of the underlying mathematical modeling. Even though the constraints of audio models are very different from those of trading systems, I still draw heavily on what I learned from my previous experience, and it has also been an opportunity to learn new things.

I guess the silver lining is that I genuinely enjoy learning. It’s possible to do so across a wide variety of domains. It just happened to be the ones you mentioned, but there are many others I’m curious about. For instance, I have a few robots at home that I experimented with at some point, but never had the time to really dig into them deeply. Too bad there are only 24 hours in a day.

Speaking of not having enough time to explore everything, what role does intuition play in your field?

Intuition plays a big role in deciding which ideas to try, but you also have to constantly integrate feedback and take recent developments in the field into account. At the same time, it’s hard to build intuition without actually trying ideas. I tend to spend more time than is probably reasonable pushing ideas to some form of proof of concept, only to end up ditching the code. As a result, I have a whole graveyard of projects that never panned out, but I always feel that I learn new things by doing this.

As an example of something random I dabbled with a few years ago that never ended up being useful, I built a small OCaml based Python interpreter that can run in the browser. You can try it here.

So many things have happened in such a short time. What has been the biggest technological breakthrough, the one in the technical field that truly blew your mind?

The combination of scale and learning in training recent foundation models. It feels incredible that we now have access to so much raw compute power through modern GPUs, and that we are able to leverage it to build truly groundbreaking technology.

You also have held many different roles, from research to engineering and now entrepreneurship. How would you define your current comfort zone?

Somewhere at the intersection of research and engineering. I’m most comfortable when I can build the technology while also influencing research directions, so that engineering and research converge toward building the best possible technology. I continue to enjoy coding on a daily basis, both for work and for personal projects.

And across these different contexts, you have written code in very different eras and domains, from formal methods to trading systems to AI research. How has your definition of “good code” evolved across these contexts?

Good code is code that works for what it was designed to do. In research, we want to be able to iterate quickly, so flexibility is key. In trading systems or when building inference platforms, we want strong performance but also exceptional robustness, which requires the best possible engineering practices.

Modern type systems are very good at providing guarantees about what can happen at runtime, and it’s both a fun and rewarding exercise to encode what you care about into the type system so that the code can be easily refactored later without worrying about things breaking.

With Moshi, we invested in building a fully custom inference platform and designing it properly from the ground up. It ended up working well, and we did not need to ship any additional releases or bug fixes after the initial release.

Many argue that software quality is going through a crisis. AI may not be the only culprit, but it is certainly part of the picture. What are your thoughts on this, and more generally, on the future of the discipline(s)?

I don’t really feel that way. There are many great open source projects that improve the quality of their code over the years, and I believe the same happens in corporate codebases. There is also a lot of lower quality code that ends up on GitHub or elsewhere, but I don’t see this as an issue. People will continue to build on solid foundations such as operating systems, compilers, ML frameworks, and so on, and the quality of these projects feels more like it is improving than anything else.

And when the field itself changes every few months, how do you make architectural decisions? What is your framework for deciding what is worth building for the long term?

The field evolves quickly, so we have to keep up with the pace, but the fundamentals actually remain similar. We are still optimizing similar model architectures, and the GPU programming model has not changed much, even if each new generation brings a few incremental improvements. To me, it mostly comes down to understanding the fundamentals well, so that it is easier to adapt to the latest developments as they occur.

His Thoughts About The Ecosystem

Many companies today are working on topics that are not really achievable yet, but eventually will be, and they want to be part of it when AGI happens for instance. Do you tend to choose topics where you believe you can make a major difference on your own, or areas that are still largely unexplored but where you expect the ecosystem to evolve alongside you?

That was the idea with Kyutai, to build and share research with the rest of the community. Having a non profit setup is great, as it makes it possible to pursue long term research without having to rush for funding.

When it comes to Gradium, we still want to maintain a culture of innovation and tackle hard and ambitious research projects, but we also have to ship products and iterate on them. So we chose a domain where we already have expertise, and where it is now very likely that research and engineering developments will work well. We want to push the boundaries of the ecosystem forward, but I don’t feel we need to go into very unexplored or highly experimental areas.

How do you personally define AGI? Do you think that this specific vision will ever be achieved? And if so, what is your ballpark estimate for when it could happen?

I don’t really have a definition of AGI, nor do I care much about it. What matters to me is that current frontier models already provide value by answering questions, helping brainstorm ideas, writing code, and so on. I think these models will continue to improve and become increasingly helpful, but I don’t see this as something with a clear AGI threshold.

Understood, so let’s take it down a notch. What do you see as the big missing pieces in AI right now, the ones that could truly change everything? Do you think we’re close to seeing them happen?

I can think of two things. First, reliability. Models have improved a lot on this over the last six months, but it is still far from perfect. Second, training efficiency. The amount of compute and data used to train models is gigantic.

It’s very impressive that we have been able to scale training this far, but it still feels like we are missing something, and that it should be possible to train good models without requiring so much. It’s hard to know what will ultimately solve this, whether it’s a better understanding of the world or something else.

And to finish on a lighter note, what is your current stack?

I’m a big advocate of functional programming. These days, I mostly work with a mix of Rust for system level aspects and distributed systems, and Python for the research side, including data analysis and writing small scripts that coordinate Rust components.