Video dubbing and subtitle engine written in Rust. Automatically transcribes, translates, and dubs videos with support for fully on-device processing on Apple Silicon via MLX. All ASR and TTS providers are free and local.

• 5-stage pipeline: Download → Transcribe → Translate → TTS → Embed subtitles
• Auto language detection: Whisper detects the source language; EN↔RU translation selected automatically
• Multi-track audio: Dubbed audio added as a second track with language metadata (original preserved)
• yt-dlp integration: Paste a YouTube URL, get a dubbed .mp4 back
• Free ASR backends: faster-whisper, whisper.cpp, WhisperKit, MLX Whisper
• Voice cloning: Clone the original speaker's voice from reference audio (Qwen3-TTS, Fish Speech, Chatterbox)
• Emotion matching: LLM-based emotion detection per subtitle, injected as inline tags for TTS providers that support them
• 5 TTS backends: Edge TTS, MLX Audio (Kokoro), Qwen3-TTS, Fish Speech S2 Pro, Chatterbox
• Any OpenAI-compatible LLM: OpenAI, DeepSeek, local LM Studio / mlx_lm.server
• On-device Apple Silicon: MLX Whisper + Qwen3-TTS + local LLM = zero cloud dependencies
• Two binaries: vdub (CLI) and vdubd (web server)

• Rust 1.75+
• macOS with Homebrew (dependencies are auto-installed on first run)

All other dependencies (ffmpeg, yt-dlp, ASR/TTS backends) are automatically installed on startup based on your config. Python packages are managed via uv in an isolated venv at ./venv/.

# Build
git clone https://github.com/jmpnop/vdub.git
cd vdub
cargo build --release

# Set up config
cp config/config-example.toml config/config.toml
# Edit config/config.toml — set your LLM API key

# Just a URL — auto-detects language, dubs, embeds subtitles
vdub https://youtube.com/watch?v=VIDEO_ID

# Specify languages
vdub https://youtube.com/watch?v=VIDEO_ID --from en --to ru

# Subtitles only, no dubbing
vdub https://youtube.com/watch?v=VIDEO_ID --no-tts

# Local file
vdub local:./my_video.mp4

# Voice-cloning TTS with emotion
vdub https://youtube.com/watch?v=VIDEO_ID --tts-provider qwen3-tts

# A/B test providers
vdub https://youtube.com/watch?v=VIDEO_ID --tts-provider fish-speech
vdub https://youtube.com/watch?v=VIDEO_ID --tts-provider chatterbox

# All options
vdub https://youtube.com/watch?v=VIDEO_ID \
  --from en --to ru \
  --tts-provider qwen3-tts \
  --no-bilingual \
  --replace-audio \
  --vertical

vdubd

Open http://127.0.0.1:8888 in your browser.

Run the entire pipeline on-device with zero cloud dependencies. Qwen3-TTS provides voice cloning + emotion matching via MLX:

# Start local LLM (LM Studio recommended, or mlx_lm.server)
# LM Studio: download Qwen2.5-72B-Instruct-abliterated, start server on port 1234

Set in config/config.toml:

[llm]
base_url = "http://localhost:1234/v1"
api_key = "not-needed"
model = "qwen2.5-72b-instruct-abliterated"

[transcribe]
provider = "mlx-whisper"

[tts]
provider = "qwen3-tts"

This runs transcription (MLX Whisper), translation (local LLM), and voice-cloned TTS (Qwen3-TTS) entirely on-device using Metal GPU.

The server accepts JSON, form-urlencoded, or just a ?url= query param:

# Simplest — just a URL
curl -d "url=https://youtube.com/watch?v=..." localhost:8888/api/capability/subtitleTask

# Or as query param
curl "localhost:8888/api/capability/subtitleTask?url=https://youtube.com/watch?v=..."

# Full JSON (all fields optional except url)
curl localhost:8888/api/capability/subtitleTask \
  -H "Content-Type: application/json" \
  -d '{"url": "https://youtube.com/watch?v=..."}'

Any OpenAI-compatible API. Point llm.base_url at your provider.

src/
  bin/          # vdub (CLI) and vdubd (web server)
  config/       # TOML config with provider enums
  dto/          # API request/response types
  handler/      # Axum HTTP handlers
  provider/     # ASR, TTS, LLM provider implementations
    openai/     # OpenAI-compatible Chat (for translation LLM)
    local/      # whisper.cpp, WhisperKit, faster-whisper, edge-tts,
                # MLX Whisper, MLX Audio, Qwen3-TTS, Fish Speech, Chatterbox
  service/      # Pipeline steps (split, transcribe, translate, TTS, embed)
  storage/      # Task store, binary path detection
  types/        # Subtitles, ASS headers, prompts, language maps
  util/         # ffmpeg/ffprobe wrappers, dependency management, CLI art,
                # voice extraction, emotion detection
scripts/        # Python TTS wrappers (qwen3_tts.py, fish_speech_tts.py, chatterbox_tts.py)
static/         # Embedded web UI
config/         # Example configuration

MIT