DuoFlow: Towards Synchronized Speech and Motion Generation with Fine-Grained Control

A dual-stream latent-flow transformer that jointly generates speech and full-body motion from a transcript, hierarchical motion conditions, and a natural-language voice description.

About the paper

Controlling how an utterance sounds and how the body moves

Abstract

An embodied speaker's expressiveness depends not only on what is said, but also on how the voice sounds and how the body moves. However, existing joint speech--motion generation models are largely driven implicitly by transcript semantics or intermediate speech representations, offering little explicit control over motion hierarchy, body parts, or vocal style. To bridge this gap, we construct MotionSpeech, a structured benchmark for controllable joint speech--motion generation that includes transcripts, voice-style descriptions, and motion conditions at two hierarchical levels. Building on this benchmark, we propose DuoFlow, a dual-stream latent flow transformer that jointly denoises speech and motion under a shared noise level and a unified second-based temporal coordinate. Modality-specific residual connectors inject structured conditions into the corresponding streams, layer-wise bidirectional cross-modal attention coordinates their generation, and a monotonic phoneme pathway combines normalized progress anchors with CTC supervision to strengthen speech temporal modeling. DuoFlow provides a unified interface for fine-grained motion and vocal-style control and demonstrates the ability to synthesize high-quality, synchronized, and instruction-consistent speech--motion sequences.

DuoFlow teaser showing transcript, hierarchical motion controls, voice description, and joint speech-motion outputs
DuoFlow overview. DuoFlow enables controllable joint generation from text conditions to speech and motion. Modifying any modality description can guide the generation: speech content (Transcript), overall motion intent (L1), fine-grained temporal motion (L2), and vocal expression (AUDCAP).

Our contributions are threefold.

Structured benchmark

We present MotionSpeech, a four-channel structured benchmark for fine-grained controllable joint generation, comprising 34,950 clips.

Unified controllable generation

We propose DuoFlow, the first joint speech–motion generation framework to support hierarchical motion and voice-condition control.

Unified evaluation

We establish a unified evaluation protocol for generation quality, cross-modal synchronization, and text–motion consistency.

MotionSpeech benchmark

Hierarchical conditions for one paired speech-motion target

MotionSpeech construction from paired SMPL-X motion and speech with L1, L2, transcript, and AUDCAP annotations
MotionSpeech construction. Motion captioning produces coarse and fine-grained descriptions; audio captioning supplies the voice description, followed by manual verification.
34,950release segments
125.2 happroximate release duration
17,907BEAT2 segments
17,043Seamless Interaction segments
4structured annotation channels
Method

Two native-rate streams, one coordinated generation process

Speech and motion preserve their native latent rates while exchanging information through gated cross-attention at every flow-transformer block.

DuoFlow architecture with structured conditions, audio and motion latent streams, phoneme conditioning, CTC supervision, and frozen decoders
DuoFlow architecture. Structured semantic conditions are adapted for both streams. The audio branch additionally receives frozen phoneme features augmented with a learned normalized progress anchor; CTC supervision is used only during training.
01

Modality-adapted conditions

Frozen Gemma features are fused and passed through separate residual connectors so the same structured prompt can be interpreted differently by speech and motion.

02

Bidirectional stream coupling

Eight dual-stream blocks combine self-attention, condition cross-attention, and gated audio-to-motion and motion-to-audio exchange on a shared temporal coordinate.

03

Phoneme-progress alignment

A normalized progress anchor injects relative phoneme order into every audio block, while multi-depth CTC supervision regularizes intermediate audio states during training.

Experiments

Current automatic benchmark results

Automatic speech-motion generation quality and alignment

DuoFlow leads on speech intelligibility, speaker similarity, motion fidelity, foot-contact stability, and both levels of PairAcc. Match-TTSG has the smallest BC Gap, showing that rhythmic alignment and motion fidelity measure different properties.

SystemWER (%) ↓SpkSim ↑FGD ↓Skate (%) ↓BC Gap ↓L1 PairAcc ↑L2 PairAcc ↑
CosyVoice2 + MambaTalk6.120.802.6146.780.2353.4253.63
Match-TTSG36.350.755.1484.010.0656.3860.38
Motion-Omni74.980.561.3266.570.3461.3762.68
DuoFlow (Ours)4.710.940.7529.610.1487.4390.40
Match-TTSG
0.35 s mean0.03 RTF1.00×
CosyVoice2 + MambaTalk
2.82 s mean0.22 RTF8.07×
Motion-Omni
5.03 s mean0.39 RTF14.39×
DuoFlow
1.33 s mean0.10 RTF3.80×
01

Joint-generation comparison

Transcript
Motion condition
Voice condition
DuoFlowOurs
▶Loading video…
CosyVoice2 + MambaTalk
▶Loading video…
Match-TTSG
▶Loading video…
Motion-Omni
▶Loading video…
02

L1 motion control

Transcript
AUDCAP
L1 conditionModified prompt
DuoFlowOurs
▶Loading video…
CosyVoice2 + MambaTalk
▶Loading video…
Match-TTSG
▶Loading video…
Motion-Omni
▶Loading video…
03

L2 motion control

Transcript
AUDCAP
L2 conditionModified prompt
DuoFlowOurs
▶Loading video…
CosyVoice2 + MambaTalk
▶Loading video…
Match-TTSG
▶Loading video…
Motion-Omni
▶Loading video…
04

AUDCAP voice control

Transcript
Motion condition
AUDCAP conditionModified prompt
DuoFlowOurs
▶Loading video…
CosyVoice2 + MambaTalk
▶Loading video…
Match-TTSG
▶Loading video…
Motion-Omni
▶Loading video…