High-fidelity zero-shot speech synthesis.

DualAR architecture

Generating a sentence runs on two clocks at once. The slow clock follows the text: it decides where one frame ends and the next begins, and keeps the sentence coherent across its whole arc. The fast clock fills each frame with timbre, articulation and spectral detail, the things that make a voice belong to one speaker. One sequence model can run at both rates, but the sequence gets needlessly long and structure and fidelity compete for the same representation.

Audio8 TTS divides speech generation into two stages using a DualAR architecture inspired by Fish Audio S2 Pro. The slower autoregressive stage plans the speech's meaning and timing, generating one semantic token for each codec frame. The faster stage then uses these hidden states, together with the codebooks already generated for the current frame, to reconstruct the fine acoustic details.

The model includes a neural audio codec that encodes an optional voice reference and decodes the generated codes into a 44.1 kHz audio waveform.

Multi-layer RVQ: multiple tokens per frame

The codec produces every frame through 10 residual codebooks of 4,096 entries each. The first few lay down the rough skeleton of the frame; later ones add residuals layer by layer, sharpening articulation, timbre and finer acoustic texture. Fast AR predicts this stack along the depth axis, so capacity goes to detail instead of the semantic stream carrying the whole waveform alone.

This is also why the two branches can work in tandem. Slow AR advances at roughly 21.5 frames per second, unrolling a stable sequence that long-range context can rely on; Fast AR then completes a much shorter within-frame decode against it. The pipeline stays end-to-end autoregressive, but the token budget follows the structure of speech instead of flattening every layer of detail into one sequence.

Audio8 TTS 0.6B

The main model has 601,159,424 parameters, not counting the neural audio codec. Both branches share a width of 896, but the depth is deliberately asymmetric: a 24-layer Slow AR carries speech content across time, while a 4-layer Fast AR concentrates on the ten codebooks inside each frame.

At 44.1 kHz, 2,048 samples is roughly 46.4 ms. Per second of audio, Slow AR makes about 21.5 semantic decisions and Fast AR fills each with ten layers of residual detail. A static KV cache keeps earlier positions available, so the model never recomputes the whole prefix at every frame.

The codec ships with the model and does double duty: the same component encodes the reference voice and decodes the generated codes. The inference path manages one component, with no separate audio checkpoint to maintain.

The 170M branch

Audio8 TTS 0.1B keeps the DualAR architecture and its existing interface, while replacing the Slow AR backbone. Instead of using a standard Transformer, it uses Falcon-H1, which combines attention layers with Mamba 2 state-space layers. This change reduces the main generative model to 170M parameters while preserving the semantic-to-acoustic handoff and the same ten codebooks.

The 170M-parameter figure applies only to the main generative model. The audio decoder is a separate component with approximately 120M parameters, and it still contributes to the total storage and memory required to generate a waveform. Reporting these components separately provides a more accurate comparison of model scale.

Falcon-H1 also changes the runtime state. Each request maintains its own FalconHybridMambaAttentionDynamicCache for the Slow AR stage, while semantic sampling, Fast AR, and audio decoding remain shared with the 0.6B model path. During GPU serving, the Mamba state runs through the eager execution path rather than the standard SGLang KV-cache pages.

A training objective that reflects the architecture

The split runs through training as well as inference. Audio is first encoded into a [10, T] array of codebook indices; supervised fine-tuning then optimizes the Slow AR semantic/EOS objective and the Fast AR codebook objective jointly under teacher forcing. When only one side of the pipeline needs to change, the other will stay frozen.

The data interface is deliberately designed small. Target audio is required; reference_audio and reference_text may be omitted, but when used they must come as a pair. At inference, text and reference audio share one packed 2,048-position context. The repository recommends keeping a single input to about 150 characters and splitting longer content at sentence boundaries.

Zero-shot voice cloning: one voice goes a long way

Voice cloning is only useful if the model can accurately separate who is speaking from what is said. Audio8 TTS never fine-tunes for a new speaker: it encodes a short recording together with its exact transcript and uses that pair as the condition for new text.

inputs = processor(
    text=["Welcome to Audio8 TTS."],
    reference_audio=["reference.wav"],
    reference_text=["The exact transcript of the reference recording."],
    return_tensors="pt",
)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.8,
        top_p=0.95,
        top_k=50,
        do_sample=True,
        return_dict_in_generate=True,
    )
    waveforms, lengths = model.decode_audio(output.codes)

The reference transcript guides generation as an input, rather than being added afterward. It aligns what was said with the reference waveform and leaves the model a cleaner speaker-identity signal. Clean recordings and accurate transcripts give the most stable results; noise, over-long clips and wrong transcripts all degrade stability and voice similarity.

The reference is optional. Omit both reference fields and the same checkpoint is an ordinary text-to-speech model.

The language changes, the voice doesn't

Audio8 TTS 0.6B covers Cantonese, Chinese, Dutch, English, French, German, Italian, Japanese, Korean, Polish and Spanish, with more being added. A Chinese reference can drive an English sentence, and an English reference can drive Chinese. The language of the text changes, but the voice ultimately stays the same.

Performance benchmark results at a compact scale

In the comparisons below, Audio8 TTS 0.6B is the smallest main model in the table. On Seed-TTS it has the lowest English WER listed and a competitive Chinese CER; on CV3 it is in the top tier of public multilingual evaluations. We make no claim of leadership on every metric or in every language.

Seed-TTS

Lower is better for WER/CER; higher is better for SIM. SIM is reported as a percentage.

CV3 Multilingual Error Rates

Lower is better for all metrics below.

The 170M model is not a same-quality substitute for the 0.6B. Its public Seed-TTS results are 1.662 WER / 56.7 SIM for English and 1.13 CER / 68.2 SIM for Chinese; on CV3 it scores 3.619 in Chinese and 3.307 in English, with wider gaps in some experimental languages. Its value is elsewhere: keeping zero-shot conditioning usable under a much smaller deployment budget.

All cross-project numbers come from public model cards and technical reports. Normalization, evaluators and test sets differ, so read these as reference comparisons, not a controlled ranking.

From checkpoint to a runnable serving stack

Parameter counts only mean something when a runtime can carry them. Audio8 TTS ships two deployment paths with different emphases: the 0.6B model goes for GPU throughput, the 170M model comes with a self-contained CPU runtime.

GPU serving for the 0.6B model

The Audio8 repository provides an SGLang Omni adapter with paged attention, dynamic batching, reference-audio encoding, waveform decoding, and an OpenAI-compatible /v1/audio/speech endpoint.

The adapter is validated against SGLang Omni 0.1.0 (commit 68a572348837f7b004857b4b07993c20ade4c017), SGLang 0.5.8, PyTorch 2.9.1+cu128, Transformers 4.57.1 and BF16 weights. In warm single-stream tests on a single NVIDIA H20, generating 128 frames takes a p50 latency of 0.691 s at an RTF of 0.116. The tests exclude cold start and compile time; these numbers describe the test configuration only, not a universal latency guarantee.

ONNX INT8: a 170M path for edge systems

The separately released Audio8-TTS-0.1B ONNX INT8 is more than a quantized checkpoint. It packages the autoregressive loop, audio codec, timbre registration and serving interfaces around ONNX Runtime — once the model is downloaded, the runtime needs no CUDA, PyTorch or Transformers.

The Slow AR graph passes attention and Mamba state explicitly between adjacent tokens. Valid-prefix attention confines compute to the valid prefix. Fast AR follows the same per-token boundary, emitting the codebooks inside each frame. Exposing the loop state explicitly is easier to schedule, inspect and memory-bound on CPU than wrapping the whole loop into one monolithic static graph.

In reported tests, regular inference occupies roughly 0.4 GiB after loading. Test platform: Linux x86_64. Memory footprint varies with ONNX Runtime build, allocator, thread count and hardware. On an 8-thread host, Slow AR takes about 19 ms per token, Fast AR about 8 ms per frame, and the one-time prefill costs roughly 19 ms per prompt token. These numbers are useful reference points; production deployments should re-measure on the target CPU.

Moving Timbre Registration off the Hot Path

Ordinary synthesis needs only Slow AR, Fast AR and the audio decoder. The audio encoder is loaded only when registering a new reference timbre, converting the recording and its transcript into reusable timbre codes; once done, the encoder can be unloaded and synthesis resumes. Timbre registration therefore becomes a one-off lifecycle operation rather than a standing cost paid on every request.

The standalone onnx_runtime_0_1b_int8 implementation includes command-line inference, a local HTTP server, streaming PCM, timbre registration and an OpenAI-compatible API. From model files to serving boundary, the deployment chain is fully covered — not just a set of exported compute graphs.

Zero-shot cloning works best with clean reference recordings and accurate transcripts, and requires the speaker's consent. Synthesized speech may be incorrectly used for impersonation or misleading content: disclose generated audio where appropriate, and evaluate accuracy, safety and legal compliance before production use.