Building a dub from its blocks

The ready dubbing workflow is made of ordinary nodes, and every one of them can be used on its own. This page builds a dub from them, one piece at a time: first the quickest dub that works, then the natural-speech rounds, then the voices, the background and the subtitles, and finally the whole engine as a sub-workflow. Each piece shows its wiring port by port — node.output → node.input — so that it can be reproduced in the editor, over the API or by an agent. Dubbing with natural speech explains why each piece exists; this page explains how it is connected.

The quickest dub: one pass, fitted by tempo

A short clip — a product video, a social reel, an announcement — can be dubbed in one pass: translate the lines, voice each one, make each voice last as long as its line, and put the voices on a timeline. No rounds, no plan:

Source Video ─→ Extract Audio ─→ Speech to Text (word timing) ─ caption_track ─→ Caption Blocks ─→ Caption Translation
                                                     └───────────── source_track ─────────────→ Aggregate Caption Track
Aggregate Caption Track.caption_track ─→ Enumerate JSON (cues: id → cue_id, text, start, end)
   ├─ text ────────→ Text to Speech ─ audio ─→ Fit Audio Duration (tempo_then_pad) ─ audio ─→ Audio Timeline.clips
   ├─ start, end ──→ Fit Audio Duration.target_start / target_end
   └─ cue_id, start, end ─────────────────────────────────────────────────────────→ Audio Timeline
Media Technical Info.duration ─→ Audio Timeline.timeline_duration ;  Audio Timeline.audio ─→ Add Audio (replace)

Fit Audio Duration (audio/fit_duration) with strategy tempo_then_pad plays a long take faster — up to Maximum Rate (maximum_rate), for example 1.25 — and pads a short one with silence; Audio Timeline (aggregate/audio_timeline) places each take at its line’s start. It costs one translation and one take per line. The price is naturalness: every long line is simply played faster, and a line 40% too long cannot fit at 1.25× at all — tempo_then_trim cuts its end, tempo_then_pad leaves it to On Out of Range (on_out_of_range), which stops the step (fail) or keeps the take longer than its line (clamp, warn). That is exactly what the natural-speech rounds avoid. Use it when the lines are short and the target language is not much longer than the source; use the rounds for anything a listener will hear at length.

Natural speech: the plan and the rounds

The natural-speech build replaces “voice once, then force it to fit” with “voice, measure, reword, voice again”. Three kinds of nodes carry it, and a timing state (madoo.speech-timing-state/v1) travels between them: the plan, the takes so far, which lines are accepted and which are still open.

The plan — Speech Timing Plan (utility/speech_timing_plan):

Aggregate Transcript.caption_track (source)  ─→ Speech Timing Plan.source_captions
Aggregate Caption Track.caption_track (target) ─→ Speech Timing Plan.target_captions
Media Technical Info.duration               ─→ Speech Timing Plan.duration

Setting	Engine value	Meaning
Use Available Caption Window (use_available_caption_window)	true	a line may use the silence after it, up to the next line
Maximum End Extension (maximum_end_extension_ms)	1000	how much of that silence, at most (1–2000 ms with the window on)
Caption Window Safety (caption_window_safety_ms)	20	the gap always left before the next line
Maximum Fit Rate / Maximum Combined Rate	2 / 2	the fastest a take may be played in round 3 (rounds 1–2 accept up to 1.10×/1.15× whatever is set)

Its outputs are the state and the pending captions — the open lines, all of them before round 1.

A round is eight nodes, always the same, fed by the previous state (the plan’s for round 1, the previous reconcile’s afterwards). Written for round N, with prev the previous state node:

prev.pending_captions ─→ Caption Blocks (duration_aware).caption_track      prev.state ─→ Caption Blocks.timing_state
Caption Blocks.block ─→ Caption Translation (duration_aware).block          Target Language ─→ Caption Translation.target_language
prev.pending_captions ─→ Aggregate Caption Track.source_track               Caption Translation.translation ─→ Aggregate Caption Track.translations
Aggregate Caption Track.caption_track ─→ Enumerate JSON (cues: id → cue_id, text → text, speaker → speaker)
Enumerate JSON.text, cue_id ─→ Text to Speech (duration_aware).text, cue_id   prev.state ─→ Text to Speech.timing_state
Target Language ─→ Text to Speech.language                                  (voices: see below)
Text to Speech.audio, generation_metadata ─→ Speech Timing Check.audio, generation
Enumerate JSON.cue_id, text ─→ Speech Timing Check.cue_id, text             prev.state ─→ Speech Timing Check.state
Speech Timing Check.candidate ─→ Aggregate JSON (column candidate: json, empty behaviour: empty)
Aggregate JSON.result ─→ Speech Timing Reconcile.candidates                 prev.state ─→ Speech Timing Reconcile.state

The reconcile’s state and pending captions feed the next round. When every line is accepted, the next round receives no pending line and is skipped at no cost; the empty collection is why Aggregate JSON must be set to produce an empty list rather than fail.

Three rounds, paid only where needed. The timing policy has three rounds, and the engine wires all three. A round costs only for the lines still open when it starts — in the W01 run, round 2 voiced 25 lines and round 3 voiced 14 of 96 — so the later rounds are cheap insurance, not a multiplier. The third reconcile has Finalize (finalize) on and Delivery Policy (delivery_policy) best_effort: it accepts every open line with its best take, reflows the ones that cannot fit, and emits the delivery outputs.

The model and the voice of the rounds. The adaptation (Caption Translation, timing mode) is a short call per line: the engine uses a fast model with max_tokens 256 and no reasoning. The first translation, which reads whole blocks, needs a larger budget — 8192 tokens. The voice model must be the same in every round, with the same settings, so that a line does not change voice between takes; the engine uses ElevenLabs Turbo v2.5, stability 0.5.

From the final state to the dubbed video

Final Reconcile.delivery_cues ─→ Enumerate JSON (array cues: cueId → cue_id, text, start, end,
      assessment.candidate.audioStorageRef → audio, .audioSha256 → expected_sha256,
      .trimStartMilliseconds → trim_start_ms, .trimEndMilliseconds → trim_end_ms, generation)
   ─→ Fit Audio Duration (natural_speech): audio, expected_sha256, trim_start_ms, trim_end_ms, start → target_start, end → target_end
   ─→ Audio Timeline: clips ← fitted audio, cue_id, start, end; timeline_duration ← Media Technical Info.duration
Audio Timeline.audio ─→ (background mix, loudness) ─→ Add Audio (replace) with the source video

Fit Audio Duration in natural_speech mode checks that the audio is the take the state chose (its SHA-256), trims its edge silence as measured, plays it faster only if it is longer than its window — pitch and formants preserved, between Minimum Rate 1 and Maximum Rate 2 — and pads the rest; Tolerance 80 ms. Audio Timeline must refuse rather than cut: Overlap Policy fail, Missing Cue Policy fail, Clip Overflow Policy fail, up to 10,000 cues.

The Dubbing Manifest (utility/dubbing_manifest, duration_aware on) takes the final state on Timing State, the collected cue evidence, the timeline manifest, the master and its technical info, the final audio and video with their measurements and quality checks, both caption tracks and, when separated, the stem evidence. It checks that every piece belongs to the same plan and the same takes before it writes a line of the manifest, and its quality check feeds a Quality Gate in warn mode.

One voice per speaker

Aggregate Transcript.caption_track ─→ List Speakers (enumerate/speakers, sample_seconds 8)
Source Video ─→ Trim Video.video      List Speakers.sample_start, sample_end ─→ Trim Video.start, end
Trim Video.video ─→ Video Analysis.video   (prompt: feminine, masculine or unclear, as JSON; temperature 0)
List Speakers.speaker ─→ Aggregate JSON.speaker (text)    Video Analysis.json ─→ Aggregate JSON.voice_type (json)
Aggregate Transcript.caption_track ─→ Speaker Voices.caption_track    Aggregate JSON.result ─→ Speaker Voices.voice_types
Speaker Voices.speaker_voices ─→ every Text to Speech.speaker_voices   each round's Enumerate JSON.speaker ─→ Text to Speech.speaker

The Speaker and Speaker Voices inputs of Text to Speech go together: one without the other is a wiring error. Voices on Speaker Voices fixes voices by speaker ({"speaker_0": "Rachel"}); Feminine Voices, Masculine Voices and Other Voices replace the lists voices are drawn from. Without the Video Analysis branch — connect nothing to Voice Types — every speaker still gets a distinct voice. The voice names must exist for the voice model: the default lists are ElevenLabs voices.

The background, optional

Source Video ─→ Extract Audio (standard, WAV) ─→ Speech-Ready Preflight.value ─→ Filter.value
Preserve Background ─→ Filter.condition
Filter.value ─→ Audio Segments (fixed_duration, 76 s, no overlap) ─→ Stem Separation (Demucs htdemucs_ft)
Stem Separation.drums + bass ─→ Audio Mix ;  + other ─→ Audio Mix ─→ Aggregate Audio Merge (the background)
Audio Timeline.audio ─→ Duck.foreground     background ─→ Duck.background (profile voice_over, background_absent_behavior pass_foreground)

Segments of 76 seconds keep each separation within the provider’s limits. The original mix reaches the Filter through the speech-ready preflight — a Quality Gate releases its value only when the transcript is usable — so nothing is separated for a video without speech. With the switch off the Filter releases nothing, the branch is skipped, and Duck, told to pass the foreground when the background is absent, delivers the dialogue alone. The manifest must then know it: its Background Mode follows the same switch.

Subtitles, optional

Two Video Subtitles nodes behind two Filters on Generate Subtitles: the source video with the source track, and the dubbed video with the final reconcile’s speech-aligned captions — the dubbed words on the times the dubbed voice takes. Their captions output is the SRT file. The engine uses the same look for both and hides speaker labels (speaker_labels hide).

The engine as a sub-workflow

The ready example keeps the engine in its own workflow and calls it from a short one with a Sub-workflow node (workflow/sub) whose Workflow is the engine’s wf_ ID. The sub-workflow’s ports are the engine’s inputs and outputs by name: an input’s name comes from its label — the engine’s input labelled “Source Video” is the port source_video — and an output’s name from its Output Name. Rename a label and the port changes with it. Keeping the engine separate lets several short workflows share it — one per market, one per client, one with the background and one without — and lets the engine be changed and republished without touching them.

The wiring rules

Validation checks a natural-speech build before anything runs and reports each problem as NATURAL_SPEECH_WIRING, naming the node and the port:

Rule	Why
Speech Timing Plan knows the master’s length: duration connected from Media Technical Info, or Duration Ms set	every timeline is exactly as long as the video
With Use Available Caption Window, Maximum End Extension is 1–2000 ms; without it, at most 250 ms and only with Reference Audio	a line may borrow silence only when the silence is known
A duration-aware Caption Blocks or Text to Speech has its Timing State connected; one that is not duration-aware does not	the round must know which lines are open and their budgets
Caption Blocks and the Caption Translation it feeds are both duration-aware, or neither	a timed block needs the timing prompt
A duration-aware Text to Speech has its Cue ID connected	each take is bound to its line
Text to Speech has both Speaker and Speaker Voices, or neither	a speaker without a map, or a map without speakers, voices nobody right
Speech Timing Check has state, audio, generation, cue_id and text, with audio and receipt from the same duration-aware Text to Speech	the measurement authenticates the voice it scores, never an edited copy
Speech Timing Reconcile has the previous state	the rounds form one chain
A duration-aware Dubbing Manifest reads Timing State from a reconcile with Finalize on, and its Audio Timeline does not use legacy_trim	the manifest describes a delivery, and a dub never silently cuts a line

What to change, and what it does

You want	Change	Effect and caution
a different voice model or voice	every round’s Text to Speech, identically	a model without a voice choice keeps one voice for everyone
a brand’s cast	Speaker Voices Voices or its lists	voices must exist for the model
more room for the lines	Plan Maximum End Extension, up to 2000 ms	longer lines may run into the next speaker’s breath; never into the next line
fewer quick lines	lower the plan’s Maximum Fit Rate and Maximum Combined Rate (1.10–2.0)	round 3 accepts less; more lines are reflowed into the pause after them
a cheaper dub	Preserve Background off	about 40% less; the dub is the dialogue alone
a glossary respected	the same Glossary on every Caption Translation, first pass and rounds	a term the rounds reword would otherwise escape it
a different mix or loudness	Duck Profile, Normalize Loudness Profile	measure again with Audio Quality Report; see Audio finishing
a dub that must never be imperfect	a Quality Gate in route mode on the manifest’s check	the dub is held back instead of delivered with warnings

Change the rates only together: the plan’s rates decide what round 3 accepts, Fit Audio Duration’s Maximum Rate decides what it can play, and the reconcile relies on both. Raising only the fit’s rate lets lines play faster without teaching the rounds to write them shorter.